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1 


Review of Probability Theory 


1.1 Elementary probability 


Definition 1.1.1. A random experiment is an experiment that can be re- 
peated under the same conditions and whose result cannot be predicted with 
certainty. 


Example 1.1.1. We consider the following three classical experiments: 

£: a coin is tossed three times and the number of “tails” obtained is recorded; 
E>: a die is thrown until a “6” appears, and the number of throws made is 
counted; 

£3: a number is taken at random in the interval (0, 1). 


Remark. A closed interval will be denoted by [a,b], whereas we write (a,b) in 
the case of an open interval, rather than Ja, b[, as some authors write. 


Definition 1.1.2. The sample space S of a random experiment is the set 
of all possible outcomes of this experiment. 


Example 1.1.2. The sample spaces that correspond to the random experiments 
in the example above are the following: 


S; = {0,1, 2,3}; 
So = {1 Qiks 
S3 = (0,1). 


Definition 1.1.3. An event is a subset of the sample space S. In particu- 
lar, each possible outcome of a random experiment is called an elementary 
event. 


The number of elementary events in a sample space may be finite ($1), 
countably infinite (S2), or uncountably infinite ($3). 
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S 


C 


Fig. 1.1. Venn diagram. 


We often use Venn! diagrams in elementary probability: the sample space 
S is represented by a rectangle and the events A, B, C, etc. by circles that 
overlap inside the rectangle (see Fig. 1.1). 


Example 1.1.3. We can define, in particular, the following events with respect 
to the sample spaces associated with the random experiments of Example 
1.1.1: 

A, = “tails” is obtained only once, that is, Ay = {1}; 

A2 = six or seven throws are made to obtain the first “6,” that is, Ag = {6,7}; 
A3 = the number taken at random is smaller than 1/2, that is, Az = (0, 1/2). 


Notations 

Union: AUB (corresponds to the case when we seek the probability that one 
event or another one occurred, or that both events occurred). 

Intersection: AN B or AB (when we seek the probability that an event and 
another one occurred). If two events are incompatible (or mutually exclusive), 
then we write that AN B = @ (the empty set). 

Complement: A‘ (the set of elementary events that do not belong to A). 
Inclusion: A Cc B (when all the elementary events that belong to A also 
belong to B). 


Definition 1.1.4. A probability measure is a function P of the subsets of 
a sample space S, associated with a random experiment E, that possesses the 
following properties: 


Axiom I: P[A]>0 VAC S; 


Axiom II: P[S] = 1; 


* John Venn, 1834-1923, was born and died in England. He was a mathemati- 
cian and priest. He taught at the University of Cambridge and worked in both 
mathematical logic and probability theory. 
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Axiom III: Jf Ay, Ao,... is an infinite sequence of events that are all incom- 
patible when taken two at a time, then 


P| U Ar = 5 PIAL (1.1) 
k=1 k=1 


In particular, 


P[AUB|=P[4]+P[B] if ANB=0 (1.2) 


If the number n of elementary events is finite and if these events are 
equiprobable (or equally likely), then we may write that 


P[A] = (1.3) 


where n(A) is the number of elementary events in A. However, in general, 
the elementary events are not equiprobable. For instance, the four elementary 
events of the sample space S, in Example 1.1.2 have the following probabil- 
ities: P[{O}] = P[{3}] = 1/8 and P[{1}] = P[{2}] = 3/8 (if we assume that 
the coin is fair), and not P[{k}] = 1/4, for k = 0,1, 2,3. 


Remark. It is said that the French mathematician d’Alembert? believed that 
when a fair coin is tossed twice, then the probability of getting one “tail” and 
one “head” is equal to 1/3. His reasoning was based on the fact that there 
are three possible outcomes in this random experiment: getting two “tails”; 
two “heads”; or one “tail” and one “head.” It is easy to determine that the 
probability of obtaining one “tail” and one “head” is actually 1/2, because 
here there are four equiprobable elementary events: H,H2 (that is, “heads” 
on the first and on the second toss); H,T2; T; H2; and TT). Finally, the event 
A: getting one “tail” and one “head” corresponds to two elementary events: 
MT, and T\A2. 


Proposition 1.1.1. We have 
1) P[AS] =1-P[A] VACS. 
2) For all events A and B, 
P{AU B] = P[A] + P[B] — P[AB] (1.4) 


3) For any three events A, B, and C, 


P[AU BUC] = P[A] + P[B] + P[C] — P[AB] — P[AC] — P[BC] + P[ABC] 
(1.5) 


? Jean Le Rond d’Alembert, 1717-1783, was born and died in France. He was a 
prolific mathematician and writer. He published books, in particular, on dynamics 
and on the equilibrium and motion of fluids. 
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Counting 


Proposition 1.1.2. If k objects are taken at random among n distinct objects 
and if the order in which the objects were drawn does matter, then the number 
of different permutations that can be obtained is given by 


nxnx...xn=nk (1.6) 


if the objects are taken with replacement, and by 


n! 
nx(n-1)x...x [n—(k -—1)] = ——— := P ork =0,1,...,n 
(1.7) 
when the objects are taken without replacement. 
Remark. If, among the n objects, there are n; of type i, where i = 1,2,...,3, 


then the number of different permutations of the entire set of objects is given 
by (the multinomial coefficients) 
n) 

malngl ny! oe 
Example 1.1.4. The combination of a certain padlock is made up of three 
digits. Therefore, theoretically there are 10? = 1000 possible combinations. 
However, in practice, if we impose the following constraint: the combination 
cannot be made up of two identical consecutive digits, then the number of 
possible combinations is given by 10 x 9 x 9 = 810. This result can also be 
obtained by subtracting from 1000 the number of combinations with at least 
two identical consecutive digits, namely, 10 (with three identical digits) + 
10x1x9+10x9x 1 (with ezactly two identical digits, either the first two 
or the last two digits). 


Proposition 1.1.3. If k objects are taken, at random and without replace- 
ment, among n distinct objects and if the order in which the objects were 
drawn does not matter, then the number of different combinations that can 
be obtained is given, fork =0,1,...,n, by 


nx(n—-1)x...x|[n—-(k-1)] n! n 
oe =C? 1.9 
ki ki(n — k)! k i oe) 
Remark. If the objects are taken with replacement, then the number of different 
combinations is C?**-!. In this case, k may take any value in N° := {0,1,...}. 


Remark. In the preceding example, according to the common use, the word 
“combinations” was used for a padlock. However, they were indeed “permu- 
tations.” 
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Example 1.1.5. In a given lottery, 6 balls are drawn at random and without 
replacement among 49 balls numbered from 1 to 49. We win a prize if the 
combination that we chose has at least three correct numbers. Let 

F = we win a prize 
and 

Fy, = our combination has exactly k correct numbers, for k = 0,1,... ,6. 
We have 


2 (7) G8) 
PIF) =1- 5° Pi) =1- 0 
k=0 k=0 (6) 
(1 - 6,096, 454) + (6 - 962, 598) + (15 - 123, 410) 
13, 983, 816 


13, 723, 192 
= 1] —- ———_—_ ~1 — 0.9814 = 0.0186 
13, 983, 816 : 


Notation. The expression P[A | B] denotes the probability of the event A, 
gwen that (or knowing that, or simply if) the event B has occurred. 


Definition 1.1.5. We set 


P[ANB} 


if P[B] > 0 (1.10) 


Proposition 1.1.4. (Multiplication rule) We have 


P[AN B] = P[A| B] x P[B] = P[B| A] x PA] if P[A|P[B] >0 (1.11) 


Example 1.1.6. In the preceding example, let 
Fj, = the number of the kth ball that is drawn is part of our combination. 
Generalizing the multiplication rule, we may write that 


PIFL ON F290 Fs] = P[F3 | Fy 9 Fo|P[F2 | Fil P[Fi] 
4.5.6 120 


ot oP ag a HOTT 
47 * 48 ~ 49 110,544 


Definition 1.1.6. The events By, Bo,...,Bn constitute a partition of the 
sample space S if 


i) BB; =O0ViFd, 
Oy Pe = 8, 
2t) P(Bg] > 0, for k = 1,2,.....,%- 
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If By, Bo,... , By is a partition of S, then we may write, for any event A, 
that 


A= (AM B,)U(AN By) U...U(AN Bn) (1.12) 


where (AN Bi) (AN B;) = OV i # j. Making use of Axiom III in the 
definition of the function P, p. 2, we obtain the following result. 


Proposition 1.1.5. (Total probability rule) If A Cc S and the events Bi, 
Bg, ..., By form a partition of S, then 


P(A] = ss P[AN Bg] = 57 PIA | By] P[ Be] (1.13) 
k=1 k=1 


Finally, we deduce from the total probability rule and from the formula 


P[B | A|P[A] 


PIA| B= 


if P[A]P[B] > 0 (1.14) 


the result known as Bayes’? rule (or formula, or also theorem). 
Proposition 1.1.6. (Bayes’ rule) If P[A] > 0, then 


P{A | Bj] P[B;] 


PIB; | Al= 55 ped | BP IBA 


for j =1,2,...,n (1.15) 


where By, Bo,...,Brn is a partition of S. 


Example 1.1.7. In a certain institution, 80% of the teaching staff are men. 
Moreover, 80% of the male teachers hold a Ph.D. and 90% of the female 
teachers hold a Ph.D. A teacher from this institution is taken at random. Let 
F = this teacher is a woman 
and 
D = this teacher holds a Ph.D. 
We may write, by the total probability rule, that 


P[D] = P[D| F|P[F| + P[D | F°|P[F‘] = 0.9 x 0.2 + 0.8 x 0.8 = 0.82 
Furthermore, we have 


prippee ee ee es aca ~ 0.2195 


* The Reverend Thomas Bayes, 1702-1761, was born and died in England. His 
works on probability theory were published in a posthumous scientific paper in 
1764. 
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Definition 1.1.7. Two events, A and B, are said to be independent if 


P[ANB] = PIAIPB] (1.16) 


Remark. Let C be an event such that P[C] > 0. We say that A and B are 
conditionally independent with respect to C if 


P[ANB|C] = P[A|C]P[B| C] (1.17) 


If, in particular, A and C, or B and C, are incompatible, then A and B are 
conditionally independent with respect to C, whether they are independent 
or not. Moreover, A and B can be independent, but not conditionally inde- 
pendent with respect to C. For instance, this may be the case if A and C, and 
B and C, are not incompatible, but AN BNC = 9. 


For events A and B such that P[A] x P[B] > 0, the next proposition could 
serve as a more intuitive definition of independence. 


Proposition 1.1.7. Two events, A and B, having a positive probability are 
independent if and only if 


P[A|B] = PIA] or P[B| A] = PIB] (1.18) 


Proposition 1.1.8. If A and B are independent, then so are A° and B, A 
and B®, and A® and BS. 


Remark. The preceding proposition is obviously false if we replace the word 
“independent” by “incompatible” (and if A and B are not the sample space 
S). 


All that remains to do is to generalize the notion of independence to any 
number of events. 


Definition 1.1.8. The events Aj, Ao,...,An are said to be independent if 
we may write that 


k 
PlA, 1A, 9... Aa] = ]] Pla] (1.19) 


j=l 


for k = 2,3,...,n, where the events A;, € {A1,...,An} Vj are all different. 


Remark. If the preceding definition is satisfied (at least) for k = 2, we say that 
the events A;, A2,...,Apn are pairwise independent. 
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ce Or 


(a) (b) 
Fig. 1.2. Examples of (a) a series system and (b) a parallel system. 


Example 1.1.8. A given system is made up of n components placed in series 
and that operate independently from one another [see Fig. 1.2 (a)]. Let 
F = the system is functioning at time to 


and 
Fy, = component k is functioning at time fo, for k =1,... ,n. 
We have % 
PIF] = PIF, FN... Fal [] PlFel 
k=1 


Remark. To help out the reader, the justification of the equality, as here by 
independence (abbreviated as ind.), is sometimes placed above the equality 
sign. 


When the components are placed in parallel, we may write that 


PI|F| =1- PIF) =1-P[Ffn...n FS 1-]Ja- 
k=1 


1.2 Random variables 


Definition 1.2.1. A random variable (r.v.) is a function X that associates 
a real number X(s) = x with each element s of S, where S is a sample space 
associated to a random experiment E. We denote by Sx the set of all possible 
values of X (see Fig. 1.3). 


Remark. The reason for which we introduce the concept of a random variable 
is that the elements s of the sample space S can be anything, for example, 
a color or a brand of object. Since we prefer to work with real numbers, we 
transform (if needed) each s into a real number x = X(s). 
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S X(s) =x 


Fig. 1.3. Graphical representation of a random variable. 


Example 1.2.1. Consider the random experiment E3 in Example 1.1.1 that 
consists of taking a number at random in the interval (0,1). In this case, the 
elements of S are already real numbers, so that we can define the r.v. X that 
is simply the number obtained. That is, X is the identity function which with 
s associates the real number s. 

We can define other random variables on the same sample space S, for 
example: 


Y= 1 if the number obtained is smaller than 1/2 
~ | 0 otherwise 


(called the indicator variable of the event A: the number obtained is smaller 
than 1/2) and Z(s) = Ins; that is, Z is the natural logarithm of the number 
taken at random in the interval (0,1). 

We have 


Sx =S=(0,1), Sy ={0,1}, and Sz = (-—o0,0) 
Definition 1.2.2. The distribution function of the r.v. X is defined by 
Fxy(z)=P[X <a} VaeR (1.20) 
Properties. i) 0 < Fx(z) <1. 
ii) limg4—oo Fx (x) = 0 and limg... Fx(z) = 1. 
iii) If zy < x2, then Fx (x1) < Fx (x2). 


iv) The function F’y is (at least) right-continuous: 


Fx (z) = Fx (at) := lim Fx (x + €) (1.21) 


Proposition 1.2.1. We have 
i) Pla <X< b| = Fx (6) - Fx (a), 
ii) PIX =a] = Fy(z) — Fx(x7), where Fx (x7) := limo Fx (z — €). 


Remark. Part ii) of the preceding proposition implies that P[X = z] = 0 for 
all x where the function Fy (x) is continuous. 
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Definition 1.2.3. The conditional distribution function of the r.v. X, 
given an event A for which P{A] > 0, is defined by 


PIX <2} nA] 


Fx(e| A) = 


(1.22) 


Remark. A marginal distribution function Fy (x) is simply the particular case 
of the preceding definition where the event A is the sample space S. 


Example 1.2.2. The distribution function of the r.v. X in Example 1.2.1 is 
given by 

Oifz<0 

Fxy(z)=P[X <az]=4 zif0<2<1 

lifz>1 
It is easy to check that this function possesses all the properties of a distribu- 
tion function. In fact, it is continuous for all real x, so that we can state that 
the number that will be taken at random in the interval (0,1) had, a priori, 
a zero probability of being chosen, which might seem contradictory. However, 
there are so many real numbers in the interval (0,1) that if we assigned a 
positive probability to each of them, then the sum of all these probabilities 
would diverge. 

Next, consider the event A: the number obtained is smaller than 1/2. Since 

P{A] = 1/2 > 0, we calculate 


Fx(z|A)=P[X <2|X <1/2)=2P[(X <a] if0<2<1/2 
so that we may write that 


0 ifx<0 
Fx (x| A) = ¢ 2x if0<a2<1/2 
1 ife>1/2 
Definition 1.2.4. If the set Sx of values that the r.v. X can take is finite or 
countably infinite, we say that X is a discrete r.v. or an r.v. of discrete 
type. 


Definition 1.2.5. The probability mass function of the discrete r.v. X is 
defined by 


px(tp) = P[X =a] V te € Sx (1.23) 


Remarks. i) Properties: a) px(z,) > 0 V zx; b) acesx px (zp) = 1. 
ii) We may write that 


Fx(2)= > px (ax) u(x ~ 2x) (1.24) 
ERESX 
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where u(x) is the Heaviside* function, defined by 


Oifa <0 


iii) Generalization: the conditional probability mass function of X, given an 
event A having a positive probability, is defined by 


PIX =2}0 Al 


Pa (1.26) 


px(x| A) = 
Example 1.2.3. An urn contains five white balls and three red balls. We draw 
one ball at a time, at random and without replacement, until we obtain a white 


ball. Let X be the number of draws needed to end the random experiment. 
We find that 


and 


Finally, let A: the first white ball is obtained after at most two draws. We 
have P[A] = Fx (2) = 3+ 2 x 3 = $8 and 


Important discrete random variables 


i) Bernoulli’ distribution: we say that X has a Bernoulli distribution with 
parameter p, where p is called the probability of a success, if 


px(x)=p*(1—p)'"* forx =O and 1 (1:27) 


* Oliver Heaviside, 1850-1925, who was born and died in England, was a physicist 
who worked in the field of electromagnetism. He invented operational calculus to 
solve ordinary differential equations. 

® Jacob (or Jacques) Bernoulli, 1654-1705, was born and died in Switzerland. His 
important book on probability theory was published eight years after his death. 


12 1 Review of Probability Theory 


Remark. The term (probability) distribution is used to designate the set of 
possible values of a discrete random variable, along with their respective prob- 
abilities given by the probability mass function. By extension, the same term 
will be employed in the continuous case (see p. 13). 


ii) Binomial distribution with parameters n and p: Sx = {0,1,... ,n} and 


px(x) = (")era aos) al (1.28) 


We write X ~ B(n,p). Some values of its distribution function are given in 
Table 6.4, p. 358. 


iii) Geometric distribution with parameter p: Sx = {1,2,...} and 


px(z) =(1—p)*"'p (1.29) 


We write X ~ Geom(p). 
iv) Poisson® distribution with parameter \ > 0: Sx = {0,1,...} and 
vz 
px() = e*— (1.30) 
x} 
We write X ~ Poi(A). Its distribution function is given, for some values of 4, 


in Table 6.4, p. 361. 


Poisson approximation. If n is large enough (>20) and p is sufficiently 
small (<0.05), then we may write that 


P[B(n, p) =k] ~ P[Poi(A = np) =k] fork =0,1,...,n (1.31) 
If the parameter p is greater than 1/2, we proceed as follows: 


P{B(n,p) = kl = P{B(n,1 - p) =n— 
~ P[Poi(A = n(1 — p)) =n— kl] (1.32) 


When p > 1/2, we also have 


P[B(n, p) < k] = P[B(n,1—p) >n—kl 
~ P[Poi(\ = n(1— p)) >n—-k] (1.33) 


§ Siméon Denis Poisson, 1781-1840, was born and died in France. He first stud- 
ied medicine and, from 1798, mathematics at the Ecole Polytechnique de Paris, 
where he taught from 1802 to 1808. His professors at the Ecole Polytechnique 
were, among others, Laplace and Lagrange. In mathematics, his main results 
were his papers on definite integrals and Fourier series. The Poisson distribution 
appeared in his important book on probability theory published in 1837. He also 
published works on mechanics, electricity, magnetism, and astronomy. His name 
is associated with numerous results in both mathematics and physics. 
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Example 1.2.4. Suppose that we repeat the random experiment in Example 
1.2.3 20 times and that we count the number of times, which we denote by 
X, that the first white ball was obtained on the fourth draw. We may write 
that X ~ B(n = 20,p = 1/56). We calculate 


20 19 
1 5) 


The approximation with a Poisson distribution gives 
P[X <1)~P{Y <1], where Y ~ Poi(20/56) 
5 
=e 5/14 4 e- 5/14 ~ 09,9496 
e +e 14 


Definition 1.2.6. A continuous random variable X is an r.v. that can 
take an uncountably infinite number of values and whose distribution function 
Fy is continuous. 


Definition 1.2.7. The (probability) density function of the continuous r.v. 
X is defined (at all points where the derivative exists) by 


fx 2) = Fx (2) (1.34) 


Remark. The function fx(z) is not the probability P[X = 2] for a continuous 
r.v. since P[X = 2] = 0 V z in this case. The interpretation that can be given 
to fx(z) is the following: 


fx(x) ~ ——-+——_—> (1.35) 


where € > 0. The equality is obtained by taking the limit as € | 0. 


Properties. i) fx(x) > 0 [by the formula (1.35), or by the formula (1.34), 
because F'x is a nondecreasing function]. 
ii) We deduce from the formula (1.34) that 


Fe(a)= f° fultyat (1.36) 
It follows that > 
[- fx(z)dz=1 (1.37) 
We also have = 
Pla < X <j = Fx(b) — Fx(a) = [ tx dir (1.38) 


Thus, the probability that X takes a value in the interval (a, }] is given by the 
area under the curve y = fx(x) from a to 0. 
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Definition 1.2.8. The conditional density function of the continuous r.v. 
X, given an event A for which P[A] > 0, is given by 


d 
fx(z| A) = GF x(z| A) (1.39) 
x 
Remark. We find that the function f(z | A) may be expressed as follows: 
fx(z) 
= 1.40 


for all x € Sx for which the event A occurs. For example, if Sx = [0,1] and 
A = {X < 1/2}, then Sy = [0,1/2), where Y := X | A. 


Remark. If X is an r.v. that can take an uncountably infinite number of 
values, but the function Fy is not continuous, then X is an r.v. of mized type. 
An example of an r.v. of mixed type is the quantity X (in inches) of rain 
or snow that will fall during a certain day in a given region. We certainly 
have the following: P[X = 0] > 0, so that X is not a continuous r.v. (since 
P[X = 0] = 0 for any continuous r.v.). It is not an r.v. of discrete type either, 
because it can (theoretically) take any value in the interval [0, 00). 


Example 1.2.5. Suppose that 


1/4 if-l<ar<1 
fx(x) = ¢ 1/(2z) ifl1<a2<e 
0 elsewhere 


We can check that the function fy is nonnegative and that its integral over 
R is indeed equal to 1. 
We calculate 


0 ifz<-l 
(e+1)/4 if-l<a<1 
(1+Inz)/2ifl<2<e 

1 ifr >e 


Fx(z) = 


Note that the density function fx(z) is discontinuous at z = 1 (and at 
x = —1 and zx = e), which is allowed, whereas the distribution function Fy is 
a continuous function, as it should be (for a continuous random variable). 

Next, we can calculate Fx(x | X < 0) and differentiate this function 
to obtain fx(z | X < 0). It is, however, more efficient to simply calculate 
P[X < 0] =1/4 and write that 


lif-l<zx#<0 
0 elsewhere 


fxle|X<0)={ 


Finally, note that the function f(z | X <0) also satisfies the two properties 
of probability density functions: it is nonnegative and its integral over R is 
equal to 1. 
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Important continuous random variables 


i) Uniform distribution on the interval [a, }: 
fx(x)=(b-—a)7! fora<ax<b (1.41) 
Notation: X ~ U[a, }}. 
ii) Exponential distribution with parameter A > 0: 
fx(a) =Ae" for x > 0 (1.42) 
Notation: X ~ Exp()). 
iti) Gamma distribution with parameters a > 0 and A > 0: 


Ne (Ano 


Tle) for x > 0 (1.43) 


fx(x) = 


where I’(a) = (a — 1)! if a € N (see p. 115). Notation: X ~ G(a, A). 


iv) Gaussian’ distribution with parameters js and 0”, where o > 0: 


fx(x) = 


=a p\2 
exp {ar | forzéR (1.44) 
270 


Notation: X ~ N(u,07). 
Remarks. i) In the particular case where js = 0 and a = 1 (see Fig. 1.4), X is 


called a standard Gaussian distribution. Its distribution function is denoted 
by @: 


z 

1 

P(z) := / ae! dy (1.45) 
~oo V2T 

The values of this function are presented in Table A.3, p. 370, for z > 0. By 

symmetry, we may write that ®(—z) = 1 — &(z). 

ii) If we define Y = aX +b, where X has a Gaussian distribution with pa- 


rameters js and o”, then we find that Y ~ N(au + 6,a?07). In particular, 
Z:=(X — p)/o ~ N(O, 1). 


Transformations. If X is a r.v., then any transformation Y := g(X), where 
g is a real-valued function defined on R, is also a random variable. 


” Carl Friedrich Gauss, 1777-1855, was born and died in Germany. He carried out 
numerous works in astronomy and physics, in addition to his important mathe- 
matical discoveries. He was interested, in particular, in algebra and geometry. He 
introduced the law of errors, that now bears his name, as a model for the errors 
in astronomical observations. 
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-1 i x 


Fig. 1.4. Standard Gaussian distribution. 


Proposition 1.2.2. Suppose that the transformation y = g(x) is bijective. 
Then the density function of Y := g(X) is given by 


fy(y) = fx (97"(y)) \z “tw (1.46) 


for y € (g(a), 9(b)] (respectively, [g(b), g(a)|) if g is a strictly increasing (resp., 
decreasing) function and Sx = {a,}]. 


Example 1.2.6. 1f X ~ U(0,1) and we define Y = e*, then we obtain 


Get: (ny) | ny - 
for y € (1,e). 


Definition 1.2.9. The mathematical expectation (or the mean) E|X] of 
the r.v. X is defined by 


E[x] = oe px(xx) (discrete case) (1.47) 
k=1 
or 
E(x] = . zfx(x)dz (continuous case) (1.48) 


Remarks. i) Generalization: we obtain the conditional (mathematical) ex- 
pectation E|X | A] of X, given an event A, by replacing px(z) by px(z | A) 
or fx(x) by fx(a | A) in the definition. 


ii) The mathematical operator EF is linear. 


Proposition 1.2.3. The mathematical expectation of g(X) is given by 
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Elg(X)] = S°g(ae)px (ae) (discrete case) (1.49) 
k=1 
or 
Elg(X)] = ie g(x) fx(x)dxz (continuous case) (1.50) 


Remark. We can calculate the mathematical expectation of g(X) by condi- 
tioning, as follows: 


mr 


Elg(X)] = }) Elg(X) | BJPIBi (1.51) 


i=l 
where B,,... ,B, is a partition of a sample space S. 


The next two definitions are particular cases of mathematical expectations 
of transformations g(.X) of the r.v. X. 


Definition 1.2.10. The kth moment (or moment of order k) of the r.v. 
X about the origin is given by E[X*], fork =0,1,2,.... 


Definition 1.2.11. The variance of the r.v. X is the nonnegative quantity 


V[X] = El(X - B[X))?| (1.52) 


Remarks. i) The standard deviation of X is defined by STD[X] = (V[X])!/?. 
The r.v. X and STD[X] have the same units of measure. 


ii) We can also calculate the variance of X by conditioning with respect to a 
partition of a sample space S,, together with the formula (1.52): 


VIX] = SD BUX — BLX))? | BPI 


i=l 


iii) Generalization: the conditional variance of X, given an event A, is de- 
fined by 


V[X | A] = E[(X — E[X | A])? | A] (1.53) 


Proposition 1.2.4. i) V[aX +b] =a?V[X] Va,bER. 
ii) V[X] = E[X?] - (E[X)). 


Example 1.2.7. The mean (or the expected value) of the r.v. Y in Example 
1.2.6 is given by 


eg | 
ay |= f yer 


We also have 
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€ 1 y? |° e? —1 
By?) = [Pod = © = 
Pye fas Z|, 5 
It follows that 
2 = =) = 
viv] = $+ ~ (e-1)? = = 4S x 0.2400 
Now, let Z := In Y. We have 
en. al (Iny)?|° 1 
BIZ) = fin ~dy=+—| =- 
[Z] jay te eee 


Note that Z is identical to the r.v. X in Example 1.2.6 and that the mean of 
X ~ U(0,1) is indeed equal to 1/2. 


A very important special case of the mathematical expectation E[g(X)] 
occurs when g(X) = e/#*., 


Definition 1.2.12. The function 
Cx (w) = Efe?”*] (1.54) 
where 7 = /—1, is called the characteristic function of the r.v. X. 


If X is a continuous r.v., then Cx(w) is the Fourier® transform of the 
density function fx (zx): 


[e@) 
Cx (w) =| eJ”* fy (x) dx (1.55) 
We can invert this Fourier transform and write that 
1 ee : 
rics / eS2Cy (ws) dew (1.56) 
27 Jo 


Since the Fourier transform is unique, the function Cx (w) characterizes en- 
tirely the r.v. X. For instance, there is only the standard Gaussian distribution 
that possesses the characteristic function Cx (w) = e~” /?. 


We can also use the function C'x(w) to obtain the moments of order n 
of the r.v. X, generally more easily than from the definition of E[X”] (for 
ne {2,3,...}). 


Proposition 1.2.5. If the mathematical expectation E |X") exists and is fi- 
nite for alin € {1,2,...}, then 


B[X"] = (—i)" 5 Cxlw)laco (1.57) 


8 Joseph (Baron) Fourier, 1768-1830, was born and died in France. He taught at 
the Collége de France and at the Ecole Polytechnique. In his main work, the 
Théorie Analytique de la Chaleur, published in 1822, he made wide use of the 
series that now bears his name, but that he did not invent. 
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Table 1.1. Means, Variances, and Characteristic Functions of the Main Random 
Variables (with q:= 1 — p) 


Distribution Variance | Characteristic function 
Bernoulli 
B(n, p) 


Geom(p) 


Poi(A) 


Ula, b] 


Exp(A) 


A= jw 


exp{jwp — $w7a7} 


Remark. Table 1.1 gives the mean, the variance, and the characteristic function 
of all the discrete and continuous random variables mentioned previously. 


Many authors prefer to work with the following function, which, as its 
name indicates, also enables us to calculate the moments of a random variable. 


Definition 1.2.13. The moment-generating function of the r.v. X is de- 
fined, if the mathematical expectation exists, by Mx (t) = Efe’*). 


Remarks, i) When X is a continuous and nonnegative r.v., Mx (t) is the 
Laplace® transform of the function fx (2). 


ii) Corresponding to the formula (1.57), we find that 


d” 
E[X"] = a Mx (lio forn = 1,2,... (1.58) 


® Pierre Simon (Marquis de) Laplace, 1749-1827, was born and died in France. In 
addition to being a mathematician and an astronomer, he was also a minister and 
a senator. He was made acount by Napoléon and marquis by Louis XVIII. He par- 
ticipated in the organization of the Ecole Polytechnique of Paris. His main works 
were on astronomy and on the calculus of probabilities: the Traité de Mécanique 
Céleste, published in five volumes, from 1799, and the Théorie Analytique des 
Probabilités, whose first edition appeared in 1812. Many mathematical formulas 
bear his name. 
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Example 1.2.8. If X ~ Poi(A), we calculate 


= a . (et A)é 
Mx(t) = S- eke AZ = er S> =e exp(e’d) 


We deduce from this formula and from Eq. (1.58) that 
E{[X] = M&(0)=A and E[X?] =)? +A 


so that V[X] = A? +.\— (A)? =2. 
Note that to obtain E[X?], we can proceed as follows: 


E[X?| = yi ae Sig a peas oo 
2 A (k- 1)! 
= a . oo - ea (re) =\+\ 
dx 2 (k—1)! an 


It is clear that it is easier to differentiate twice the function e~* exp(e’A) than 
to evaluate the infinite sum above. 


When we do not know the distribution of the r.v. X, we can use the 
following inequalities to obtain bounds for the probability of certain events. 


Proposition 1.2.6. a) (Markov’s!® inequality) If X is an r.v. that takes 
only nonnegative values, then 


Pix >¢q < 24 ¥e>0 (1.59) 
b) (Chebyshev’s! inequality) Jf E[Y] and V[Y] exist, then we have 


PIlY - E[Y]|>d <4 Ve>0 (1.60) 


V[Y] 
C2 
10 Andrei Andreyevich Markov, 1856-1922, who was born and died in Russia, was a 
professor at St. Petersburg University. His first works were on number theory and 
mathematical analysis. He proved the central limit theorem under quite general 
conditions. His study of what is now called Markov chains initiated the theory of 
stochastic processes. He was also interested in poetry. 

™! Pafnuty Lvovich Chebyshev, 1821-1894, was born and died in Russia. By using 
the inequality that bears his name, he gave a simple proof of the law of large 
numbers. He also worked intensively on the central limit theorem. 
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1.3 Random vectors 


Definition 1.3.1. An n-dimensional random vector is a function X = 
(X1,...,Xn) that associates a vector (X1(s),... ,Xn(s)) of real numbers with 
each element s of a sample space S of a random experiment E. Each compo- 
nent X;, of the vector is a random variable. We denote by Sx (C R”) the set 
of all possible values of X. 


Remark. As in the case of the random variables, we will use the abbreviation 
r.v., for random vector, since there is no risk of confusion between a random 
variable and a random vector. 


Two-dimensional random vectors 


Definition 1.3.2. The joint distribution function of the r.v. (X,Y) is 
defined, for all points (x,y) € R?, by 


Fyy(2,y) = PIX <2} {Y < ys PIX <2,Y <ul (1.61) 
Properties. i) Fx,y(—00, y) = Fx,y(z, -—00) = 0 and Fx,y (00, 00) = 1. 


ii) Fx y(21,y1) < Fx y (x2, yo) if ry < xg and yr < ye. 
iii) We have 


pi Fx y(x + €, y) = im Fx y (2, yt é) = Fx y (a, y) (1.62) 


Remark. We can show that 


Pla <X<bc<Y< d| = Fy y (0, d) — Fx y (b,c) — Fx y(a,d) + Fx y (a,c) 
(1.63) 


where a, b,c, and d are constants. 


It is easy to obtain the marginal distribution function of X when the 
function F'y,y is known. Indeed, we may write that 


Fx (x) = P[X <2,Y < oo] = Fx y(z, 00) (1.64) 


Definition 1.3.3. A two-dimensional r.v., Z = (X,Y), ts of discrete type 
if Sz is a finite or countably infinite set of points in R?: 


Sz =Sxyy = {(2;,¥k).J = Pe Qari ;k= 1 eee (1.65) 


Definition 1.3.4, The joint probability mass function of the discrete r.v. 
(X,Y) is defined by 


Px,Y (25, YK) = P[X =2;,Y = ysl (1.66) 
for j,k =1,2,.... 
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To obtain the marginal probability mass function of X, from px,y, we 
make use of the total probability rule: 


px (aj) = P[X = aj] = S> PX = 2s} {¥ =ye}] =D pxy (2s, e) 
(1.67) 


Example 1.3.1. The generalization of the binomial distribution to the two- 
dimensional case is the joint probability mass function given by 


! 
m 21,09 


7 Pit p2?(1 — pi — p2)™ "2"? (1.68) 


£1,2 SMe hi ee ee 
Pal 1) 2) Ly!axo!(m — 2, — £2 )! 


where 11,22 € {0,1,...,m} and 21 +22 <mEN. We say that (X),X2) has 
a trinomial distribution with parameters m, pi, and po, where 0 < px < 1, for 
k = 1,2. We can generalize further the binomial distribution and obtain the 
multinomial distribution (in n dimensions). 


Definition 1.3.5. A two-dimensional r.v., Z = (X,Y), is of continuous 
type if Sz is an uncountably infinite subset of R?. (We assume that X and 
Y are two continuous random variables.) 


Remark. We will not consider in this book the case of random vectors with at 
least one component being a random variable of mixed type. 


Definition 1.3.6. The joint (probability) density function of the con- 
tinuous r.v. Z = (X,Y) is defined by 


fxy(z,y) = aa F xy (2,9) (1.69) 


a 
OxOy 
for any point where the derivative exists. 


Remarks. i) Corresponding to the formula (1.67), the marginal (probabil- 
ity) density function of X can be obtained as follows: 


oo 
7 i: fx,y(@,y) dy (1.70) 
—0O 
where we integrate in practice over all the values that Y can take when X = x. 


ii) The probability of the event {Z € A}, where A C Sz, can be calculated as 
follows: 


P[Z € Al =f | tev y) dady (1.71) 
A 


iii) The distribution function of the continuous r.v. (X,Y) is given by 
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Fx y(2,y) ie i” Fx,y (u,v) dudu (1.72) 
In the discrete case, the formula above becomes 
Fyy(zy) = >> Y= px.y (a5, Yr) (1.73) 
eySe YR SY 


Example 1.3.2. The continuous r.v. (X,Y) has the joint density function 


l 
wt fi<e<e0<cy<n 
fxy(z,y)=¢ 7 


0 elsewhere 


We calculate I 
= [ dy = Ina ifl<a<e 


and 

Ing In* x 1 

SE pa =- fO<y<l1 

1 x v 2 1 2 : y 
rath € _ Th? 

fy(y) = [= _In‘s 1 ae ee ee 

i 2 ; 2 
0 elsewhere 


Example 1.3.3. Let 


_ f2ye* ife>0,0<y<1 
fev ea) = { 0 elsewhere 


First, we calculate 


1 
fx(x) = / 2ye"*dy=e" ifx>0 
0 
and 
CO 
fy(y) -{ Qye "dx =2y if0<y<1 
0 
Moreover, we find that 
0 ife<Oory <0 


Fy y(z,y) = § (l-e7*)y? ifa >O0and0<y<1 
e* ifa >Oandy>1 
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Finally, we calculate 
1 l-y 
PIX+¥>1)=1-Pixtystai- ff 2ye—” dady 
0 Jo 


1 
= 1- | 2y (1 — e4!) dy =1- (1 —2e7!) = 2e7! 
0 


because ; ‘ 
i ye? dy = yerls - f e’dy=e-—(e-1)=1 
0 0 


Definition 1.3.7. Let (X,Y) be a random vector. We say that X and Y are 
independent random variables if 


Px,Y (£3, Yn) = px (xj)py (yx) if (X,Y) és discrete (1.74) 
or 
fx,y(@,y) = fx(z)fy(y) if (X,Y) ts continuous (1.75) 
Remarks. i) More generally, X and Y are independent if (and only if) 
P[X € A,Y € B] = P[X € AJP[Y € B} (1.76) 


where A (respectively, B) is any event that involves only X (resp., Y). In 
particular, we must have 


Fy y(z,y) = PIX <2,Y sy) =Fx(z)Fy(y) V (#,y) (1.77) 


ii) If X and Y are independent r.v.s, then so are g(X) and A(Y). 


iii) Let Z := X + Y, where X and Y are two independent r.v.s. Then [see the 
formula (1.102)], 


Mz(t) = Ele’?] = Efe(X+¥)] = Efe'*|E[e*Y] = Mx(t)My(t) (1.78) 
Similarly, Cz(w) = Cx(w)Cy(w). 


Example 1.3.4. We deduce from Eq. (1.75) that the r.v.s X and Y in Example 
1.3.2 are not independent, whereas those in Example 1.3.3 are. 


Definition 1.3.8. If Y is a discrete r.v., then the conditional distribution 
function of X, given that Y = yx, is defined by 


PIX $2,Y = ys) 


PIY = yal fP[Y=y)>0 — (1.79) 


Fyyy(z | yk) = 


Remark. In theory, Y can be a continuous r.v. in the preceding definition. 
However, in practice, most of the time the two random variables in the pair 
(X,Y) are of the same type. 


1.3 Random vectors 25 


Definition 1.3.9. If (X,Y) is a discrete r.v., then the conditional proba- 
bility mass function of X, given that Y = yx, is defined by 
_ Dx,y (Xj, Yk) = P[X = Dir = Yk] 


Pxiy (23 | Ye) = =e Pea if P[Y = yx] > 0 
(1.80) 


Remark. The conditional functions possess the same properties as the corre- 
sponding marginal functions. 


When Y is a continuous r.v., we cannot condition on the event {Y = y} 
directly, because P[Y = y| = 0 for all y. We must rather take the limit as 
dy decreases to zero of the functions defined by conditioning on the event 
{y <Y <y+dy}. We then obtain the following proposition. 


Proposition 1.3.1. If (X,Y) is a continuous r.v. and fy(y) > 0, then the 
conditional distribution function and the conditional density function 
of X, given that Y = y, are given, respectively, by 


x , d 
Fxyy(e |y) = Seer and frev(« | y) = BX (1.81) 


Example 1.3.5. The conditional density function of Y, given that X = 2, in 
Example 1.3.2 is 


fxy (x,y) ee 
v) = 2 = - if0<y<er 
That is, ¥Y | {X = x} has a uniform distribution on the interval (0, z). Hence, 
we easily find that Fy;x(y| 2) =O ify <0, 


Fyix(y| 2) = 2 if0<y<z 


and Fyix(y|z)=lify> cz. 


Proposition 1.3.2. The r.v.s X and Y are independent if and only if the 
conditional distribution function, the conditional probability mass function, 
or the conditional density function of X, given that Y = y, is identical to the 
corresponding marginal function. 


Example 1.3.6. We say that the continuous random variables X and Y have 
a binormal (or bivariate normal) distribution with parameters ux € R, 
by € R, of > 0, of > 0, and p € (—1,1), and we write that (X,Y) ~ 
N(ux, by; o%, 0%; p), if their joint density function is 


1 


fx.y(,y) = Inoxoy(l— Py 
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x0 = aaa SE) + (EG) 


= ape = bate] ! (1.82) 


Oxoy 


for (x,y) € R?. 
We easily find that X ~ N(yx,0%) and Y ~ N(py,0%,). It follows that 


X | {Y =y} ~ N(ux + p(ox/oy)(y — py), 0% (1 — p”)) (1.83) 


Since X and X | {Y = y} have the same distribution if p = 0, we can state 
that X and Y are independent r.v.s if the parameter p is equal to zero. This 
parameter is actually the correlation coefficient of X and Y. 


Definition 1.3.10. The conditional expectation of X, given that Y = y, 
is defined by 


oO 
E[X |Y =y]= S >a; pxiy(z;|y) (discrete case) (1.84) 
j=l 
or 
oo 
E[X |Y =y]= / rfxjy(x|y)dx (continuous case) (1.85) 


The mean E[g(X)] of a transformation g of a random variable X is a real 
constant, while E[g(X) | Y = y] is a function of y, where y is a particular 
value taken by the r.v. Y. We now consider E[g(X) | Y]. It is a function of 
the r.v. Y that takes the value E[g(X) | Y = y] when Y = y. Consequently, 
E{g(X) | Y] is a random variable, whose mean can be calculated. We then 
obtain the following important proposition. 


Proposition 1.3.3. We have 
Elg(X)] = E[E|9(X) | Yj (1.86) 
Remarks. i) We deduce from the preceding proposition that 


re ELX | Y = yl py (ye) (discrete case) 
E[X]=EIEIX|Y]=4 
Jo, E[X | Y = y] fy (y) dy (continuous case) 
(1.87) 


ii) We can calculate the variance of X by conditioning on another r.v. Y as 
follows: 
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V[X] = B(B[X? | ¥]] - (EIBLX | YJ)? (1.88) 


iii) Let X 1, Xo,... be r.v.s that possess the same distribution as the r.v. X, 
so that E[X;,] = E[X] and V[X;] = V[X], for k = 1,2,..., and let N be an 
r.v. independent of the X;’s and taking its values in the set {1,2,...}. By 
making use of the formula (1.86), we can show (see p. 254) that 


N 
E bs x = E[NJE[X] (1.89) 
k=1 
If the r.v.s X, are independent among themselves, we also have (see p. 254) 
N 
dL Xe 
k=1 


V = E(N]V[X] + V[N](£[X])? (1.90) 


iv) Suppose that we wish to estimate a random variable X by using another 
r.v. Y. It can be shown that the function g(Y) that minimizes the mean-square 
error (MSE) 


-MSE := E[(X — g(Y))?| (1.91) 


is 9(Y) = E[X | Y]. If we look for a function of the form g(Y) = aY + @, we 
can show that the constants a and @ that minimize MSE are 


E[XY] — E[X]EIY] 


VIF] and 6 = E[X]-4ElY] (1.92) 


a= 
Finally, if g(Y) = c, we easily find that the constant c that yields the smallest 
MSE is c= E[X]. 


The function g(Y) = E[X | Y] is the best estimator of X, in terms of Y, 
while g(Y) = GY + is the best linear estimator of X, in terms of Y. If X 
and Y both have a Gaussian distribution, then the two estimators are equal 
(see Ex. 1.3.6). 


Proposition 1.3.3 also enables us to calculate the probability of the event 
{X € A} by conditioning on the possible values of an r.v. Y. We only have 
to define the r.v. W such that W =1 if X € A and W =0 if X ¢ A, and use 
the fact that E[W] = P[X € A]. We can then show the following proposition, 
which is the equivalent of the total probability rule for random variables. This 
proposition and Proposition 1.3.3 will be very useful in the next chapters. 


Proposition 1.3.4. We may write that 


we PIX € ALY = yx] py (yx) (discrete case) 
PIX € Al = (1.93) 
i ease P[xX €A|Y =y] fy(y)dy (continuous case) 
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Corresponding to the definition of the conditional variance V[X | A] that 
was given in the preceding section, we now have the following definition. 


Definition 1.3.11. The conditional variance of X, given the r.v. Y, is 
defined by 


VIX | ¥] = El(X - E[X | ¥)? 1 ¥] (1.94) 
Remarks. i) We find that 

VIX | ¥] = LX? | Y] - (EIX | Y))? (1.95) 
ii) We can show the following useful result: 

V[X] = E[V[X | Y]]} + V[E[X | Y]] (1.96) 
Example 1.3.7. Instead of calculating the variance of the r.v. Y in Example 
1.3.2 from the density function fy that we obtained and from the definition of 


V[Y], we can use the fact (see Ex. 1.3.5) that Y | X ~ U(0,X). It follows that 
E{Y | X] = X/2 and V[Y | X] = X?/12, and then, by the formula (1.96), 


viY] = = EX? + *V(X] = 1 EX?) = 1 ELX])? ~ 0.4252 
12 4 3 4 
because 


e e : 1 
Bx] = | ninede = 5(¢? +1) and EX? = f x Ina dx = 5(2e° +1) 


Remark. We can check that 


E{Y] = =(e? +1) and BY?| = (26 +1) 


Remark. We can consider conditional expectations E|X | Aj, or conditional 
density functions fx y(x,y | A), etc., with respect to more general events 
A, like Y < y, Y > 0, etc., and also with respect to events A that involve 
both random variables, X and Y. For instance, let X; and X2 be independent 
random variables having a U(0,1) distribution. We have 


PIX, <2 | Xy < X94] 


PIX, < 21,X1 < X9] i 
cama «15, COS 25 ars < xy, Xe 2X5 | Xo Hao] 1d 
Pi Xs < Xa) 2) P(X: <21,X1 < X2| Xo = 29] -1dzrz 


s 1 1 
pt 2f PIX: $01,X1 <aa|dz=24 [ adr + [ 11 dea} 
0 x 


= x1(2 = 21) 
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so that 
fx, (v1 | Xi < X92) = 2(1 — 121) forO<24,< 1 


Next, we have 


1 

1 

BLK |X <Xal= f x, :2(1—2,) dry = = 
0 


Actually, if we are only looking for the mean of Xj, given that X; < Xo, we 
can directly write that 


1 
EIX, |X) < Xo] = i iE B15 dade = 5 


where we used the formula 


Fx; ,Xq (1,22) 


for 0 < < <1 
Pixie xs oo ee 


fx1,X2(1, 02 | X1 < X2) = 


The following proposition is the two-dimensional version of Proposition 
1.2.2. 


Proposition 1.3.5. Let W := g:(X,Y) and Z := go(X,Y), where X and Y 
are two continuous r.v.s. If 

1) the system w = gi (x,y), z = go(z,y) has the unique solution x = h(w,z), 
y = ho(w,2) 

and 

2) the functions g, and g2 have continuous partial derivatives V (x,y) and the 
Jacobian J(x,y) of the transformation is such that 


O91 O92 Og2 Ogi 


qe oe a? Oe By. 


#0 V (2,y) (1.97) 
then 
fw,z(w, 2) = fx y(hi(w, z), he(w, z)) {J (hi (w, z), ho(w,z))|" (1.98) 


Remarks. i) The proposition can be easily generalized to the n-dimensional 
case, where n € {3,4,... }. 

ii) In the particular case where X and Y are independent and Z := X+Y, we 
could use the proposition to obtain the density function of Z. We must first 
define an appropriate auciliary variable W, then calculate the joint density 
function of the r.v. (W,Z), and finally integrate this joint density function 
with respect to w to obtain fz(z). We can also proceed as follows: 


=f fie Pyne eee ee y= fi txlw nea 


(1.99) 


Note that the density function of Z is the convolution product of the density 
functions of X and Y. 
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Proposition 1.3.6. The mathematical expectation of the random variable 
Z :=G(X,Y) is given by 


jet 2 kn1 9(2 js Yk) PX,Y (15, Yk) (discrete case) 
E{Z] = (1.100) 


i. Nias g(x,y) fx,y(z,y) dx dy (continuous case) 
Remark. If the mathematical expectations E[X] and E[Y] exist, we have 
ElaX + bY] =aE[X]+bE[Y] Va,beER (1.101) 
Moreover, if X and Y are independent r.v.s and g(X,Y) = 91(X)go(Y), then 
Elg(X,Y)] = Elgi(X)JElg2(¥)| (1.102) 
Definition 1.3.12. The covariance of X and Y is defined by 


Cov[X, Y] = E[(X — E[X])(Y — E[Y])] = E[XY] — E[X]E[Y] (1.103) 


Remarks. i) The covariance generalizes the variance, since Cov[|X, X] = V[X], 
but the covariance Cov[X, Y] can be negative. For example, if Y = —X, then 
we have 


Cov[X, Y] = Cov[X, —X] = E[X(—X)] — E[X]E[-X] = -V[X] <0 
(1.104) 


ii) We deduce from Eq. (1.102) that if X and Y are independent, then 
Cov[X, Y] = 0. However, the converse is not always true. 


iii) We also define the correlation coefficient of X and Y by 


Cov[X, Y] 


ox  SED(X/STDY] a 


We then deduce from Example 1.3.6 that, in the case of the bivariate normal 
distribution, the r.v.s X and Y are independent #f and only if their correlation 
coefficient is equal to zero. 


An important particular case of transformations of random vectors is the 
one where the random variable Z := g(X1,... , Xn) is a linear combination of 
the r.v.s X1,...,Xn: 

Z=09 +a,X,4+-++ + OnXn (1.106) 


where the a,’s are real constants V k. We can show the following proposition. 
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Proposition 1.3.7. Let Z be a linear combination of the r.v.s X1,...,Xn- 
We can write (if the mathematical expectations exist) that 


E[Z] = a9 + 1 E[X1] +--+ + Gn E[ Xz] (1.107) 
and 
n n n 
V[Z] = So aV [Xa] +25— S- ajaxCov[X;, Xe] (1.108) 
k=1 j=lk=1 
——’” 
j<k 


Ezample 1.3.8. If X and Y are two independent r.v.s having a uniform distri- 
bution on the interval [0,1] and Z:= X + Y, then Sz = [0, 2] and 


foe) 1 
falz)= f fx (u) fy (2—u)du= [ fy (2-1) du 


Since 


lifz-l<u<z 
0 elsewhere 


fr(e-w={ 
we may write that 


Zz 
i Idu=z if0<z<l 
0 


a 1 
fa (2) = / ldu-2-zifl<z<2 
z—l 


0 elsewhere 


Remark. If we define the auxiliary variable W = X, then we find that 
fwz(w,z)=1 if0<w<10<z<2,w<z<wHl 


Integrating fw.z(w,z) with respect to w, we retrieve the function fz (z) 
above. 


Next, if Z:= X — } and W := Z?, we calculate 
Cov[Z, W] = E[ZW] — E[Z]E[W] = E[Z*] -0 =0 
because E[Z?**+1] = 0, for all k € {0,1,...}. However, Z and W are not 


independent, since Z = 0 > W = 0, in particular. 
Finally, Eq. (1.108) enables us to write that 
2 
1 
5 ee 
(x-2) | 


V[Z - 3W] = V[Z] + 9V[W] - 6Cov[Z, W] =V x- 5| +9V 
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11 
=V[X]+9V[T?], where T~U 5: | 


2 2 
1 1 2 
-5+°(m)- 4 


because 


1/2 1/2 2 
V(r?) = E(t) - (E[T?))? = / t*-1dt— i t?-1dt 
~1/2 -1/2 
Pee Oe ee ee ee 
~ 5x16 \3x4/ ~~ 80x 144 180 
We continue with limit theorems that will be used in the subsequent chap- 


ters. 


Proposition 1.3.8. Let X1,X2,... be an infinite sequence of independent 
and identically distributed (i.1.d.) r.v.s, and let S, i= X1 +++: + Xn. 


a) (Weak law of large numbers) Jf E[X;] = € R, then 


no 


lim P |S =n) <e] =1 Ve>o0 (1.109) 
b) (Strong law of large numbers) If E[X?] < 00, then we may write that 


P | iim: 2 =1 =1 (1.110) 
now nN 

c) (Central limit theorem) If E[Xi] = p € R and V[X\] = o? € (0,00), 

then we have 


lim P ee < | = P[N(0,1) < 2] (1.111) 


noo 


Remarks. i) Actually, the condition E[X?] < 00 is a sufficient condition for 
the strong law of large numbers to hold. It may be replaced by the weaker 
condition E||X1|] < oo, which reduces to E[.X,] < oo in the case when X > 0. 


ii) The central limit theorem (CLT) implies that 


Sy» N(np,no*) and = w= N(p,07/n) (1.112) 


In general, from n = 30, the approximation by the Gaussian distribution 
should be rather good. 
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iii) An application of the CLT is the Moivre!?-Laplace Gaussian approxima- 
tion to a binomial distribution: 


P[B(n, p) =k] ~ fz(k), where Z ~ N(np, np(1 — p)) (1.113) 


For the approximation to be good, the minimum between np and n(1 — p) 
should be at least equal to 5. 


Example 1.3.9. lf X; ~ Exp(1), for k = 1,... ,30, and if the X,’s are inde- 
pendent r.v.s, then we can show that 


S39 = X1 +... + X39 ~ G30, 1) 
Making use of the formula 
P{G(n, A) < 2] = P[Poi(Ax) > n] (1.114) 


we obtain (from a table of the distribution function of the Poisson distribution) 
that 
P[S39 < 30] = P[Poi(30) > 30] ~ 0.5243 


The approximation by the CLT yields 
P[S39 < 30] ~ P[N(30, 30) < 30] = 0.5 


Example 1.3.10. Suppose that 1% of the tires manufactured by a certain com- 
pany do not conform to the norms (or are defective). What is the probability 
that among 1000 tires, there are exactly 10 that do not conform to the norms? 


Solution. Let X be the number of tires that do not conform to the norms 
among the 1000 tires. If we assume that the tires are independent, then 
has a binomial distribution with parameters n = 1000 and p = 0.01. We seek 


P(X = 10] = fz(10), where Z ~ N(10,9.9) 
eee ee 1 (10 — 10)? 
~ VarJ99 Pl 2 9.9 


Remarks. i) In fact, we obtain that PLX = 10] ~ 0.1257. By using the Poisson 
approximation (see p. 12), we find that 


\ ~ 0.1268 


12 One of the pioneers of the calculus of probabilities, Abraham de Moivre, 1667- 
1754, was born in France and died in England. The definition of independence of 
two events can be found in his book The Doctrine of Chance published in 1718. 
The formula attributed to Stirling appeared in a book that he published in 1730. 
He later used this formula to prove the Gaussian approximation to the binomial 
distribution. 
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P[X = 10] ~ P[Poi(10) = 10] ~ 0.1251 


In this example, the Poisson approximation is slightly more accurate. However, 
if we increase the value of the probability p, the Moivre-Laplace approxima- 
tion should be better. 


ii) To calculate approximately a probability like P[5 < X < 10], we would 
rather use the distribution function of the Gaussian distribution. It is then 
recommended to make a continuity correction to improve the approximation. 
That is, we write that 


PIB< X < 10) = Pl4.5 < X < 10.5] ~ P[4.5 < Z< 10.5] 


1.4 Exercises 


Section 1.1 


Question no. 1 

In urn A, there are four red balls and two white balls, while urn B contains 
two red balls and four white balls. We throw, only once, a coin for which the 
probability of “tails” is equal to p (0 < p <.1). If we get “tails,” then we will 
draw balls from urn A; otherwise, urn B will be used. 
(a) What is the probability of obtaining a red ball on any draw? 


(b) If we obtained a red ball on each of the first two draws, what is the 
probability of obtaining a red ball on the third draw? 


(c) If we obtained a red ball on each of the first n draws, what is the probability 
that we are using urn A? 


Question no. 2 

Box 1 contains 1000 transistors, of which 100 are defective, and box 2 
contains 2000 transistors, of which 100 are also defective. A box is taken 
at random and two transistors are drawn from it, at random and without 
replacement. 


(a) Calculate the probability that both transistors are defective. 


(b) Given that both transistors are defective, what is the probability that they 
come from box 1? 


Question no. 3 

Assume that there is a leap year every four years. How many (independent) 
persons must be in a room, at minimum, if we want the probability that at 
least one of these persons was born on February 29 to be greater than 1/2? 
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Question no. 4 

Object O; moves on the z-axis between 0 and 2, while object Oz moves 
on the y-axis between 0 and 1. Suppose that the position of each object is 
completely random. What is the probability that the distance between the 
two objects is greater than 1? 


Question no. 5 

A certain user of the public transport system can take bus no. N; or bus 
no. N2 to go to work. A bus no. N; runs near his home every hour, from 6:00 
a.m., while a bus no. No runs there d minutes after the hour, where d & (0, 30]. 
The user arrives at the bus stop at a completely random time between 7:45 
a.m. and 8:15 a.m. What is the value of d if he takes a bus no. N; thrice more 
often than a bus no. No? 


Question no. 6 

Two sport teams play a series of (independent) games to win a trophy. The 
first team that wins four games gets the trophy. There are no draws. What is 
the probability that a team having, for each game, only a one-in-three chance 
of winning gets the trophy? 


Question no. 7 
In how many ways can we permute the numbers 1,2,... ,n if we do not 
want a single number to remain in its original position? 


Question no. 8 

In the dice game called craps, the player tosses two (fair) dice simultane- 
ously. If the sum of the two numbers that show up is equal to 7 or 11, the 
player wins. If the sum is equal to 2, 3, or 12, he loses. When the sum is a 
number z different from the preceding numbers, the player must toss the two 
dice again until he gets a sum equal to x or 7. If x is obtained first, the player 
wins; otherwise, he loses. What is the probability that the player wins? 


Question no. 9 

A says that B told her that C has lied. If the three persons tell the truth 
and lie with probability p € (0,1), independently from one another, what is 
the probability that C’ has indeed lied? 


Question no. 10 

A man takes part in a television game show. At the end, he is presented 
with three doors and is asked to choose one among them. The grand prize is 
hidden, at random, behind one of the doors, while there is nothing behind the 
other two doors. The game show host knows where the grand prize has been 
hidden. Suppose that the man has chosen door no. 1 and that the host tells 
him that he did well in not choosing door no. 3, because there was nothing 
behind it. He then offers the man the opportunity to change his choice and, 
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therefore, to select. door no. 2 instead. What is the probability that the man 
will win the grand prize if he decides to stick with door no. 1? 


Section 1.2 


Question no. 11 

Boxes I and II both contain n transistors. At each step, a fair coin is tossed. 
If “heads” (respectively, “tails”) is obtained, we take, at random and without 
replacement, a transistor in box I (resp., II). We repeat this experiment until 
one of the two boxes is empty. Let N be the number of transistors that remain 
in the other box at that moment. If we assume that the repeated trials are 
independent, what is the probability mass function of N? 


Question no. 12 
Let X be a continuous random variable whose density function is given by 


fx(z)=c’ae"* for xz >0 


where ¢ is a positive constant. Calculate E[X | X < 1]. 


Question no. 13 

A mathematician hesitates between three methods to solve a certain prob- 
lem. With the first (respectively, second) method he will work in vain for two 
(resp., three) hours, while the third method will give him the solution at 
once. If we assume that at each step the mathematician uses a method taken 
at random among those that he still has not tried, what is the variance of the 
number of hours that he will have to work to solve his problem? 


Question no. 14 
Let X be a random variable whose moment-generating function, 44x (t), 
exists for t € (—c,c). Show that 


P[X >a) <e"“Mx(t) for0O<t<c 
and 

P[X <a) <e"“Mx(t) for -c<t<0 
where a is a real constant. 


Question no. 15 
Suppose that the moment-generating function of X exists for every real 
value of t and is given by 


—et. 
1 ift =0 


Mx(t) = 
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Use the results of the preceding question to show that 


P[X >1)=0 and P[X <-1]=0 


Question no. 16 

Let X be a continuous random variable whose set of possible values is the 
interval [a,b]. We define Y = g(X). 
(a) Calculate the probability density function of Y if g(x) = 1— F(a). 
Indication. The inverse distribution function Fy’ exists. 
(b) Find a transformation g(x) such that 


1 
AW =% forl<y<3 


Question no. 17 
Calculate E[X | X > 1], where X is a random variable having a standard 
Gaussian distribution. 


Question no. 18 
Two players, X and Y, take turns at tossing a fair coin. The first one that 
gets “tails” wins. Calculate, assuming that X starts, 


(a) the probability that X wins, 


(b) the probability that X wins, given that she did not obtain “tails” on her 
first two trials, 


(c) the average number of tosses needed to end the game, given that X lost. 


Question no. 19 
Suppose that the probability that a family has exactly n children is given 
by 
P,=cp" forn=1,2,... 


where c > 0 and0<p<1l,and R=1- eae P,,. Suppose also that every 
child is equally likely to be a male or a female. 


(a) Calculate the probability that a family with n children has exactly & male 
children, for k = 0,1,... ,n. 


(b) Find the probability that a family has no male children. 
(c) What is the average number of male children per family? 


Question no. 20 

A box contains 200 brand A and 10 brand B transistors. Twenty transistors 
are taken at random. Let X be the number of brand A transistors obtained. 
(a) Calculate P[X = 20], assuming that the transistors are taken without 
replacement. 
(b) Calculate P[X = 19] if the transistors are taken with replacement. 
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(c) Use a Poisson distribution to calculate approximately P|X = 18] when 
the transistors are taken with replacement. 


Question no. 21 
The density function of the random variable X is given by 


_ fk-—2*)if0<r<1 
fx(a) = { 0 elsewhere 


where k is a positive constant. 

(a) Calculate fx (x | X? < 1/4). 

(b) Find the constant b that minimizes E[(X — 6)?). 
(c) Find the constant c that minimizes E[|.X — cl]. 


Indication. The value zm for which F’x(%m) = 1/2 is called the median of the 
continuous r.v. X. It can be shown that, for any real constant a, 


Ellx — al] = Elle — tml) + 2 | (eae) ae 
a 
Question no. 22 
We say that the continuous random variable X , whose set of possible values 


is the interval (0,00), has a Pareto!* distribution with parameter @ > 0 if its 
density function is of the form 


0 ; 
f(a) = payer N=" 


0 elsewhere 


In economics, the Pareto distribution is used to describe the (unequal) dis- 
tribution of wealth. Suppose that, in a given country, the wealth X of an 
individual (in thousands of dollars) has a Pareto distribution with parameter 
9 = 1.2. 


(a) Calculate fx(2|1<X < 3). 
(b) What is the median wealth (see Question no. 21) in this country? 
(c) We find that about 11.65% of the population has a personal wealth of at 


least. $5000, which is the average wealth in this population. What percentage 
of the total wealth of this country does this 11.65% of the population own? 


Question no. 23 
Let 
_ fkare-*/? if >0 
fx(2) = 0 elsewhere 


'3 Vilfredo Pareto, 1848-1923, born in France and died in Switzerland, was an 
economist and sociologist. He observed that 20% of the Italian population owned 
80% of the wealth of the country, which was generalized by the concept of Pareto 
distribution. 
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where & is a positive constant. 
(a) Calculate fx (x | X < 1.282). 


Indication. We have P[N(0,1) < 1.282] ~ 0.9. 
(b) Find the value of rp for which Fy (rq) ~ 0.35. 


Remark. The random variable X defined above actually has a Marwell' dis- 
tribution with parameter a = 1. In the general case where a > 0, we may 
write that 


fx(z) = +3 - 2/7 92¢-2*/(2a*) ifx>0 
a 


We find that E[X] = 2a,/2/m and V[X] = a?[3 — (8/m)]. This distribu- 
tion is used in statistical mechanics, in particular, to describe the velocity of 
molecules in thermal equilibrium. 


Question no. 24 
Let X be a continuous random variable having the density function 


He 27/20") 

—e ifr >0 
x(x) = 4 @ 
fx(q) 0 elsewhere 


We say that X has a Rayleigh'® distribution with parameter 6 > 0. 

(a) Show that E[X] = 0,/7/2 and V[X] = 6? [2 — (/2)]. 

(b) Let Y := In X, where X has a Rayleigh distribution with parameter @ = 1. 
Calculate (i) fy (1) and (ii) the moment-generating function of Y at t = 2. 


(c) We define Z = 1/X. Calculate the mathematical expectation of Z if @ = 1 
as in (b). 


Section 1.3 


Question no. 25 

The lifetime X (in days) of a device has an exponential distribution with 
parameter A. Moreover, the fraction of time during which the device is used 
each day has a uniform distribution on the interval [0, 1], independently from 
one day to another. Let N be the number of complete days during which the 
device is in a working state. 


(a) Show that P[N > n] = (1 — e7*)"/d", for n =1,2,.... 


14 James Clerk Maxwell, 1831-1879, was born in Scotland and died in England. He 
was a physicist and mathematician who worked in the fields of electricity and 
magnetism. 

8 John William Strutt Rayleigh, 1842-1919, was born and died in England. He 
won the Nobel Prize for physics in 1905. The distribution that bears his name 
is associated with the phenomenon known as Rayleigh fading in communication 
theory. 
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Indication. Because of the memoryless property of the exponential distribu- 
tion, that is, 


PIX >s+t|X>t]=P[X>s] Vs,t2>0 


it is as if we started anew every day. 
(b) Calculate E[N | N < 2] if \ = 1/10. 


Question no. 26 
Let X, and Xz be two independent N(u, 07) random variables. We set Yi 
= Xy + X_ and Yo = X +2Xo. 
(a) What is the joint density function of Y; and Y2? 
(b) What is the covariance of Y; and Yo? 


Question no. 27 

Let X, and X2 be two independent random variables. If X; has a gamma 
distribution with parameters n/2 and 1/2, and Y := X; + X2q has a gamma 
distribution with parameters m/2 and 1/2, where m > n, what is the distri- 
bution of X9? 
Indication. If X has a gamma distribution with parameters a and X, then (see 
Table 1.1, p. 19) 

rN 
Cc = ——_—_——_ 
d= Baye 

Question no. 28 

Show that if E[(X — Y)?] = 0, then P[X = Y] = 1, where X and Y are 
arbitrary random variables. 


Question no. 29 
The conditional variance of X, given the random variable Y, has been 
defined (see p. 28) by 


V[X | Y] = EX - E[X | Y]? | ¥] 
Prove the formula (1.96): 
V[X] = E[V[X | Y]] + V{E[X | Y]] 
Question no. 30 
Let X be a random variable having a Poisson distribution with parameter 


Y, where Y is an exponential r.v. with mean equal to 1. Show that W := X+1 
has a geometric distribution with parameter 1/2. That is, 


pwn) = 1/2)” for n= 1,2,... 
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Question no. 31 
Let X, and X2 be two independent random variables, both having a stan- 
dard Gaussian distribution. 


(a) Calculate the joint density function of Y; := X? + X3 and Yo := Xo. 
(b) What is the marginal density function of Y;? 
Question no. 32 
Let X and Y be independent and identically distributed random variables. 
(a) Show that h(z) := E[X | X +Y =z] =2/2. 
(b) Evaluate E[(X — h(Z))?] in terms of V[X]. 
Question no. 33 


A company found out that the quantity X of a certain product it sells 
during a given time period has the following conditional density function: 


4 
fxiy(z|y) = ge ifx>0 


where Y is a random variable whose reciprocal Z := 1/Y has a gamma dis- 
tribution. That is, 


—rz Ce ae 


eleite Saerae > 
T(a) for z >0 


fa(z) = r«€ 
(a) Obtain the marginal density function of X. 
(b) Calculate E[Z | X = a]. 
Question no. 34 


Let X and Y be continuous and independent random variables. Express 
the conditional density function of Z := X + Y, given that X = 2, in terms 


of fy. 


Question no. 35 
Show that for continuous random variables X and Y, we have 


BY |X <al=ao5 ft ElY |X =ulfx(u)du 


if Fx(x) > 0. 


Question no. 36 
Let X1,...,X, be independent random variables such that 


an/T 
fx, (2) = zt? + a? 
for —co < 4% < 00 and k =1,2,... ,n, where ay > OV k. 


(a) Calculate the density function of the sum Z := X,+...+Xn. 
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Indication. The characteristic function of X, is given by 
Ox, (w) = eral 


(b) Assuming that a, = a;, for k = 2,3,..., can we state that fz(z) tends 
toward a Gaussian probability density? Justify. 


Question no. 37 

Suppose that X ~ N(0,1) and Y ~ N(1,1) are random variables such that 
pPx,y = p, where px.y is the correlation coefficient of X and Y. Calculate 
E|X?Y?]. 
Question no. 38 


Let f 
an eYVifaz > 0, y >= 
fxy (x,y) = { 0 elsewhere 


be the joint density function of the random vector (X,Y). 


(a) Find the estimator g(Y) of X, in terms of Y, that minimizes the mean- 
square error MSE := E[(X — g(Y))?]. 


(b) Calculate the minimum mean-square error. 


Question no. 39 
The joint density function of the random vector (X,Y) is given by 


3(7? 4+ cy + y?) if -1<a2<land-l<y<1 
18 
fy (ay) { 0 elsewhere 


Calculate (a) E[X | Y = 1/2], (b) E[Y | X], (c) the mean-square error MSE 


:= E[(Y — g(X))*] made by using g(X) := E[Y | X] to estimate the random 
variable Y. 


Indication. It can be shown that 


MSE = E[Y?] — Elg?(X)|_ if g(X) = E[Y | X] 


Question no. 40 

A number X is taken at random in the interval [0,1], and then a number 
Y is taken at random in the interval (0, X]. Finally, a number Z is taken at 
random in the interval [0, Y]. Calculate (a) E[Z], (b) V[Y], (c) P[Z < 1/2]. 
Question no, 41 

An angle A is taken at random in the interval [0, 7/2], so that 


fa(a) = 2 forO<a<a/2 
T 
Let X := cos A and Y := sin A. Calculate 


(a) PX =1|Y=0], (b) E[Y |X], (c) E[Y], (d) E[X | X +Y], 
(ec) E[X?|X+Y], ( B[X| Al, (eg) PIX =0| {X = O}U {X = v3/2}]. 
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Indication. We have 


d 1 
— arccosf = —~-——=—=—__ for -l<ax<l 
dx 


V1 — x? 
(h) V[X | Y] if the angle A is taken at random in the interval (0,7). 


Question no. 42 
Let 
fxiyy(e@ly)=ye"™™ forr >0,0<y<1 


Calculate, assuming that Y has a uniform distribution on the interval (0, 1), 
(a) fx(z), (b) P[XY > 1], (c) V[X|Y], (d) E[X]. 


Question no. 43 

We suppose that the (random) number N of customers that arrive at 
an automatic teller machine to withdraw money, during a given hour, has 
a Poisson distribution with parameter \ = 5. Moreover, the amount X of 
money withdrawn by an arbitrary customer is a discrete random variable 
whose probability mass function is given by 


px (x) = : if z = 20, 40, 60, 80, or 100 
Finally, we assume that N and X are independent random variables. Let Y 
be the total amount of money withdrawn over a one-hour period. Calculate 


(a) E[Y|N>0], (b) P[Y =60], (c) PIN=3|Y = 60]. 


Question no. 44 

Let X; and X2 be two independent random variables having a uniform 
distribution on the interval (0,1). We define Y = max{X1, X2}. Calculate (a) 
Fyix,(y| a1), (b) E[Y | X1= 2], (c) VIE[Y | Xi]], (a) B{V[Y | Xi]). 
Indication. If X is a nonnegative continuous (or mixed type) random variable, 
then 


EIX] = fo <Pyta)\de 


Question no. 45 

We consider a system made up of two components placed in parallel and 
operating independently. That is, both components operate at the same time, 
but the system functions if at least one of them is operational. Let T; be the 
lifetime of component i, for i = 1,2, and let T be the lifetime of the system. 
We suppose that T; ~ Exp(1/2), for i = 1,2. Calculate 
(a) EIT j= 1], (b) E(T |\T > 1], (c) E[T | {T, > 1} U {Tz > 1}]. 


Indication. In (c), we can use the formula 


E(X | AUB] =E[X | A] P[A| AUB] + E£[X | B] P[B| AUB] 
— E[X | ANB] P[ANB|AUB] 
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Question no. 46 
Let X; ~ U[-1, 1], X2 ~ U0, 1], and X3 ~ U[0, 2] be independent random 
variables. Calculate 
xX 
(a) PIX < Kale (b) PL <2 < Xb (0 BLA], (@) BZ |X), 
where Y := X1 + X9 and Z := X, + X3. 


Question no. 47 
Suppose that X1 and Xo are independent random variables such that 


fx,(vi) = dre! for ay ER 


where A; is a positive constant, for i = 1,2. Calculate 

(a) P[X1 < Xa], (b) V[X1 | X1 > 0}, (ce) El}Xa] | |X] >, 
(d) E|X + Xo | X1< XQ] if Ay = Ao. 

Question no. 48 


Calculate P[Y > X] if X ~ B(2,1/2) and Y ~ Poi(1) are two independent 
random variables. 


Question no. 49 
Use the central limit theorem to calculate (approximately) 


PIX, +...4 X49 < Xa +... + X10] 


where Xj,...,X199 are independent random variables, each having a U[0, 1] 
distribution. 


Question no. 50 
Suppose that the random variables X,,..., X39 are independent and all 
have the probability density function 


fx (x) == forl<ax<e 


What is the approximate density function of the product X1X2--- X30? 


Question no. 51 

Let (X,Y) be a random vector having a bivariate normal distribution. 
(a) Calculate P[XY < 0] if wx =0, py =0, 0% =1, 0? =4, and p= 0. 
(b) What is the best estimator of X? in terms of Y when px = 0, py = 0, 
o% =1, o? =1, and p= 0? 
(c) Calculate E[XY] when px = 1, py = 2, 0% =1, of =4, and p = 1/2. 
Question no. 52 


Let X and Y be two random variables, and let g and h be real-valued 
functions. Show that 
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(a) Elg(X) | X] = 9(X), 
(b) Blg(X)A(Y)] = E [h(¥)Elg(X) | Y]}. 


Question no. 53 


Letters are generated at random (among the 26 letters of the alphabet) 
until the word “me” has been formed, in this order, with the two most recent 
letters. Let N be the total number of letters that will have to be generated to 
end the random experiment, and let X; be the kth generated letter. It can be 
shown that E{[N] = 676 and V[N] = 454,948. Calculate (a) E[N | X2 = el, 
(b) EN | X, = mij, and (c) E[N? | X,; =m]. 

Remark. The variable X;, is not a random variable in the strict sense of the 
term, because its possible values are not real numbers. We can say that it is an 
example of a qualitative (rather than quantitative) variable. It could easily be 
transformed into a real random variable by defining X;, instead to be equal to 
j if the kth generated letter is the jth letter of the alphabet, for 7 = 1,... , 26. 


Question no. 54 
Let X;, for i = 1,2,3, be independent random variables, each having a 
uniform distribution on the interval (0,1). Calculate 
(a) E[X, + Xo + X3 | X1 + Xo], 
(b) E[X, + $Xo2 | Xi + X2+ Xz], 
(c) EV [xy | xX + X92 + X3]]. 
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Stochastic Processes 


2.1 Introduction and definitions 


Definition 2.1.1. Suppose that with each element s of a sample space S of 
some random experiment E, we associate a function X(t,s), where t belongs 
toT CR. The set {X(t,s),t € T} is called a stochastic (or random) 
process. 


Remarks. i) The function X(t, s) is a random variable for any particular value 
of t. 


ii) In this book, the set T will generally be the set N° = {0,1,...} or the 
interval [0, 00). 


Classification of the stochastic processes 


We consider the case when T is either a countably infinite set or an uncount- 
ably infinite set. Moreover, the set of possible values of the random variables 
X(t,s) can be discrete (that is, finite or countably infinite) or continuous 
(that is, uncountably infinite). Consequently, there are four different types of 
stochastic processes (s.p.). 


Definition 2.1.2. IfT is a countably infinite set (respectively, an interval or 
a set of intervals), then {X(t,s),t € T} is said to be a discrete-time (resp., 
continuous-time) stochastic process. 


Remarks. i) Except in Section 2.3, it will not be necessary to write explicitly 
the argument s of the function X(t,s). Thus, the stochastic process will be 
denoted by {X(t),t € T}. However, in the discrete case, it is customary to 
write {X,,n € T}. 

ii) We will not consider in this book the case when T is the union of a set of 
points and of an uncountably infinite set. 
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Fig. 2.1. Example of a random walk. 


Example 2.1.1. A classic example of a stochastic process is the one where we 
consider a particle that, at time 0, is at the origin. At each time unit, a coin 
is tossed. If “tails” (respectively, “heads” ) is obtained, the particle moves one 
unit to the right (resp., left) (see Fig. 2.1). Thus, the random variable X, 
denotes the position of the particle after n tosses of the coin, and the s.p. 
{X,,n = 0,1,...} is a particular random walk (see Chapter 3). Note that 
here the index n can simply denote the toss number (or the number of times 
the coin has been tossed) and it is not necessary to introduce the notion of 
time in this example. 


Example 2.1.2. An elementary continuous-time s.p., {X (£),t > 0}, is obtained 
by defining 
X(t)=Yt fort >0 


where Y is a random variable having an arbitrary distribution. 


Definition 2.1.3. The set Sx (4) of values that the r.v.s X(t) can take ts called 
the state space of the stochastic process {X(t),t € T}. If Sx) is finite or 
countably infinite (respectively, uncountably infinite), {X(t),t € T} is said to 
be a discrete-state (resp., continuous-state) process. 


Example 2.1.3. The random walk in Example 2.1.1 is a discrete-time and 
discrete-state s.p., since Sx, = {0,+1,+2,...}. For the continuous time s.p. 
in Example 2.1.2, it is a continuous-state process, unless Y takes on the value 
0, because Sx (4) = (0,00) if Y > 0 and Sx (4) = (—00, 0] if Y < 0. 


As we mentioned above, for any fixed value of t, we obtain a random 
variable X(t) (= X(t,s)). Although many authors use the notation X(t) to 
designate the stochastic process itself, we prefer to use the notation {X(t),t € 
T} to avoid the possible confusion between the s.p. and the random variable. 
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Definition 2.1.4. The distribution function of order k of the stochastic 
process {X(t),t € T} ts the joint distribution function of the random vector 
(XG) aes »X(th)): 


F(ay,... ,Upjt1,.-- tk) = P[X(ti) <a1,...,X (tk) < xy] (2.1) 


Similarly, we define the probability mass and density functions of order 
k of an s.p.: 


PCC. Bee a) Se a Se = el (2.2) 
and (where the derivative exists) 


oF 
CF (tise Bhi tty te) (2.3) 


Lycee p LE Le te) = 
f( 1 tk, e1, , tr) Oz1...02 


Remark. When k = 1 or 2, the preceding definitions are in fact only new 
notations for functions already defined in Sections 1.2 and 1.3. 


Example 2.1.4. If the tosses of the coin are independent in Example 2.1.1, 
then we may write, with p := P[{Tails}], that the first-order probability mass 
function (or probability mass function of order 1) of the process at time n = 2 
is given by 


2p(1 — p) ifx =0 


2 ifv=2 
p(z;n =2)=P[X2.=2] = apy? ife = —2 
0 otherwise 


First- and second-order moments of stochastic processes 


Just like the means, variances, and covariances enable us to characterize, 
at least partially, random variables and vectors, we can also characterize a 
stochastic process with the help of its moments. 


Definition 2.1.5. The mean E[X(t)] of ans.p. {X(t),t € T} at teme t ts de- 
noted by mx(t). Moreover, the autocorrelation function and the autoco- 
variance function of the process at the point (t;,t2) are defined, respectively, 
by 


Rx (t1, te) = E|X(t1)X (te)| (2.4) 
and 
Cx(t1,t2) = Rx (ti, t2) — mx (ti1)mx (te) (2.5) 
Finally, the correlation coefficient of the s.p. at the point (t1,t2) is 


Cx (t1, ta) (2.6) 


nt cee 
PO) [Cx (t1, t1) Cx (ta, ta) ]*/? 
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Remarks. i) In the case of pairs of random variables, the quantity E[XY] 
is called the correlation of the vector (X,Y). Here, we use the prefix “auto” 
because the function is calculated for two values of the same stochastic process 
{X(t),t € T}. The function Ry y(t1,t2) = E[X(t1)Y(t2)], where {Y(t),t € 
T*} is another s.p., is named the cross-correlation function, etc. In fact, we 
could simply use the term correlation function in the case of the function 
Rx (ty, te). 

ii) The function Rx (t,t) = E[X?(t)| is called the average power of the stochas- 
tic process {X(t),t € T}. Furthermore, the variance of the process at time t 
is 


V[X(t)] = Cx(t,t) (2.7) 


Since V[X(t)] > 0, we then deduce from Eq. (2.6) that px (t,t) = 1. 


Two properties of stochastic processes that will be assumed to hold true 
in the definition of the Wiener! process (see Chapter 4) and of the Poisson? 
process (see Chapter 5), in particular, are given in the following definitions. 


Definition 2.1.6. If the random variables X(t4) — X(t3) and X(t2) — X(t1) 
are independent V t; < te < t3 < t4, we say that the stochastic process 
{X(t),t € T} is a process with independent increments. 


Definition 2.1.7. If the random variables X (to +s) — X(t1 +s) and X(t2) - 
X (ti) have the same distribution function for all s, {X(t),t € T} is said to 
be a process with stationary increments. 


Remarks. i) The random variables X (tz +s) — X(t,+s) and X(tz2) ~ X (#1) in 
the preceding definition are identically distributed. However, in general, they 
are not equal. 


ii) The Poisson process is a process that counts the number of events, for 
instance, the arrival of customers or of phone calls, that occurred in the in- 
terval [0,¢]. By assuming that this process possesses these two properties, we 
take for granted that the r.v.s that designate the number of events in disjoint 
intervals are independent, and that the distribution of the number of events 
in a given interval depends only on the length of this interval. In practice, 
we can doubt these assertions. For example, the fact of having had many (or 


1 Norbert Wiener, 1894-1964, was born in the United States and died in Sweden. 
He obtained his Ph.D. in philosophy from Harvard University at the age of 18. 
His research subject was mathematical logic. After a stay in Europe to study 
mathematics, he started working at the Massachusetts Institute of Technology, 
where he did some research on Brownian motion. He contributed, in particular, 
to communication theory and to control. In fact, he is the inventor of cybernet- 
ics, which is the “science of communication and control in the animal and the 
machine.” 

2 See p. 12. 
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very few) customers on a given morning should give us some indication about 
how the rest of the day will unfold. Similarly, the arrival rate of customers 
at a store is generally not constant in time. There are rush periods and slack 
periods that occur at about the same hours day after day. Nevertheless, these 
simplifying assumptions enable us, for instance, to obtain explicit answers to 
problems in the theory of queues. Without these assumptions, it would be 
very difficult to calculate many quantities of interest exactly. 


Example 2.1.5. Independent trials for which the probability of success is the 
same for each of these trials are called Bernoulli trials. For example, we can 
roll some die independently an indefinite number of times and define a success 
as being the rolling of a “6.” 

A Bernoulli process is a sequence X,, X2,... of Bernoulli r.v.s associated 
with Bernoulli trials. That is, X; = 1 if the kth trial is a success and X, = 0 
otherwise. We easily calculate 


E(|X;] =p Vke {1,2,...} 
where p is the probability of a success, and 


2 if k k 
Rx (ki, k2) = E[Xe, Xk] = i; if i as 


It follows that Cx(k,, ke) =0 if ky # kg and Cx(k,,k2) = p(1—p) if ky = ko. 
Example 2.1.6. Let Y be a random variable having a U(0, 1) distribution. We 
define the stochastic process {X(t),t > 0} by 

X(t)=et fort >0 


The first-order density function of the process can be obtained by using Propo- 
sition 1.2.2 (see Example 1.2.6): 


Plast) = fxcy(e) = fy (In(a/t)) a | 


1 
=-— if x € (t,te) 
£ 


Next, the mean E|[X(t)| of the process at time t > 0 is given by 


E[X(t)] = [ ett-1dy=we-1 for t >0 


or, equivalently, by 


te 
Exo] = | n= dz=te-~t=te-1) ore 0 
t 
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Finally, we have 
X(t)X(t +s) =e't-eY (t+) =e?" t(t+s) 
Tt follows that 


e? — 


Rx(t,t+s) = E[X(t)X(t + s)] = Ele?’ t(t + s)] = t(t + 8) V s,t>0 


2.2 Stationarity 


Definition 2.2.1. We say that the stochastic process {X(t),t € T} is sta- 
tionary, or strict-sense stationary (SSS), if its distribution function of 
order n is invariant under any change of origin: 


FGijoei tap tiyess ote) =F Cie ee (onpti tf Sis yt F3) (2.8) 
for alls, n, and t),... ,tn. 


Remark. The value of s in the preceding definition must be chosen so that 
te +se€T, fork =1,...,n. So, if T = [0, 00), for instance, then t, +s must 
be nonnegative for all k. 

In practice, it is difficult to show that a given stochastic process is station- 
ary in the strict sense (except in the case of Gaussian processes, as will be 
seen in Section 2.4). Consequently, we often satisfy ourselves with a weaker 
version of the notion of stationarity, by considering only the cases where n = 
1 and n = 2 in Definition 2.2.1. 

If {X(t),t € T} is a (continuous) SSS process, then we may write that 


f(z;t) = f(ajt+s) Vast (2.9) 
and 
f (v1, 29; ti,t2) = f(@1, ro3t) + 8,te +8) V s,t1,to (2.10) 


We deduce from Eq. (2.9) that the first-order density function of the process 
must actually be independent of t: 


f(a;t)=f(x) Vt (2.11) 


Moreover, Eq. (2.10) implies that it is not necessary to know explicitly the 
values of ty and tg to be able to evaluate f(x, 2;ti,t2). It is sufficient to 
know the difference ty — t): 


f (21, £23 t1,t2) = f(ai,2aj3te —ti) V ti, te (2.12) 


In terms of the moments of the process, Eqs. (2.11) and (2.12) imply that 
mx(t) is a constant and that the autocorrelation function Rx(t1,¢2) is, in 
fact, a function RX of a single variable: Rx (ti,t2) = RX(t2 —t1). By abuse 
of notation, we simply write that Rx (ti, te) = Rx (te —t1). 
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Definition 2.2.2. We say that the stochastic process {X(t),t € T} is wide- 
sense stationary (WSS) if mx(t) =m and 


Rx(ti,te) =Rx(te-t1) Vti,to€eT (2.13) 


Remarks. i) Since mx(t) = m if {X(t),t € T} is wide-sense stationary, we 
can also write that 


Cx (#1, tz) = Rx (ti, te) ~ mx (t1)m-x (t2) = Rx (to _ ty) —-m= Cx (ta —t1) 
(2.14) 


V ti,te € T. Similarly, we have 


Cx (te — t1) 


ew (2.15) 


px (ti, te) = px(te —t) = 
ii) By choosing t; = tz = t, we obtain that E[X?(t)] = Rx(t,t) = Rx (0), for 
all t € T. Therefore, the average power of a WSS s.p. does not depend on t. 
iii) We often take t; = t and tg = t-+s when we calculate the function Rx (or 


Cx). If the process considered is WSS, then the function obtained depends 
only on s. 


iv) It is clear that an SSS stochastic process is also WSS. We will see in Section 
2.4 that, in the case of Gaussian processes, the converse is true as well. 


Example 2.2.1. The most important continuous-time and continuous-state 
stochastic process is the Wiener process, {W(t),t > 0}, which will be the 
subject (in part) of Chapter 4. We will see that E[W(t)] = 0 and that 


Cw(t,t +s) = Rw(t,t+s) =07t 


where og > 0 is a constant and s,t > 0. Since the function Rw(t,t+s) depends 
on t (rather than on s), the Wiener process is not wide-sense stationary. 


Example 2.2.2. The Poisson process, that we denote by {N(t),t > 0} and that 
will be studied in detail in Chapter 5, possesses the following characteristics: 


E[N(t)]}=At and Ry(ti,t2) = A min{t1, to} 


for all t, t1, and t2 > 0, where A is a positive constant. It is therefore not 
stationary (not even in the wide sense), because its mean depends on t. If we 
define the stochastic process {X(t),t > 0} by 


x() =" for t > 0 


then the mean of the process is a constant. However, we find that we can- 
not write that Rx(ti,t2) = Rx(t2 —t,). Consequently, this process is not 
stationary either. 
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Remark. By definition, the Wiener and Poisson processes have stationary in- 
crements. However, as we have just seen, they are not even wide-sense sta- 
tionary. Therefore, these two notions must not be confused. 


Example 2.2.3. An elementary example of a strict-sense stationary stochastic 
process is obtained by setting 


X(t)=Y fort >0 


where Y is an arbitrary random variable. Since X(t) does not depend on the 
variable t, the process {X(t),t > 0} necessarily satisfies Eq. (2.8) in Definition 
2.2.1. 


Definition 2.2.3. The spectral density of a wide-sense stationary sto- 
chastic process, {X(t),t € T}, is the Fourier transform Sx(w) of its auto- 
correlation function: 


Sx(w) = i * 6-38 Ry (5) ds (2.16) 


—oo 
Remarks. i) Inverting the Fourier transform, we obtain that 


Rx(s) os / ss 3 Sx (w) dw (2.17) 


~ OF =e 


ii) Since the autocorrelation function of a WSS process is an even function 
(that is, Rx(—s) = Rx(s)), the spectral density Sx(w) is a real and even 
function. We can then write that 


oO 1 Coo 
Sx(w) = | Rx(s)coswsds and Rx(s) = -{ Sx (w) cosws dw 
0 0 
(2.18) 
iti) It can be shown (the Wiener-Khintchin® theorem) that the spectral den- 


sity Sx (w) is a nonnegative function. Actually, a function Sx (w) is a spectral 
density if and only if it is nonnegative. 


Suppose now that the following relation holds between the processes 
{X(t),t € T} and {Y(t),t € T}: 


Y(t) = X(t) * A(t) = a X(t — s)h(s) ds (2.19) 


3 Aleksandr Yakovlevich Khintchin, 1894-1959, was born and died in Russia. He 
contributed in a very important way to the development of the theory of stochas- 
tic processes. He was also interested in statistical mechanics and in information 
theory. 
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Remark. We can interpret the process {Y(t),t € T} as being the output of 
a linear system whose input is the process {X(t),t € T}. We write Y(t) = 
L{X(#)]. 

We assume again that {X(t),t € T} is stationary in the wide sense. If 
E[X(t)] = 0, we can show that {Y(t),t € T} is a WSS process with zero 
mean and such that 


Sy (w) = Sx (w)|H(w) |? (2.20) 
where 
H(w) := ‘- e I”8h(s) ds (2.21) 
We also have ~ 
Ely?(t)] = Ry (0) -5f Sx(w)|H(w)|? dw (> 0) (2.32) 


2.3 Ergodicity 


In statistics, to estimate an unknown parameter of a distribution function, for 
example, the parameter A of an r.v. X having a Poi(A) distribution, we draw 
a random sample of X. That is, we take n observations, X1,...,Xn, of X 
and we assume that the X;’s have the same distribution function as X and 
are independent. Next, we write that the estimator \ of \ (which is the mean 
of the distribution) is the arithmetic mean of the observations. Similarly, to 
estimate the mean mx (t) of a stochastic process {X(t),t € T} at time t, we 
must first take observations X(t, s,) of the process. Next, we define 


i n 
a = (t, 2.23 
mx(b) = aA Sk) (2.23) 


Thus, we estimate the mean mx (t) of the s.p. by the mean of a random sample 
taken at time ¢. Of course, the more observations of the process at time t we 
have, the more precise the estimator m mx (6) ) should be. Suppose, however, that 
we only have a single observation, X(t, s1), of X(t). Since we cannot estimate 
mx(t) in a reasonable way from a single observation, we would like to use 
the values of the process for the other values of t to estimate m x(t). For this 
to be possible, it is necessary (but not sufficient) that the mean mx(t) be 
independent of t. 


Definition 2.3.1. The temporal mean of the s.p. {X(t),t € R} is defined 
by 


Ss 
(X(t))s = = / (tsa (2.24) 
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Remarks. i) In this section, we will suppose that the set T' is the entire 
real line. If T = {0,00), for example, we can modify the definition above. 
Moreover, in the discrete case, the integral is replaced by a sum. Thus, when 
T = {0,+1,+2,...}, we can write that 


1 N 
(Xp)n c= at oe Xn,(s) (2.25) 


where JN is a natural number. 


ii) We call a realization or trajectory of the process {X(t),t € T} the graph 
of X(t, s) as a function of t, for a fixed value of s. 


Definition 2.3.2. The stochastic process {X(t),t € T} ts said to be ergodic 
if any characteristic of the process can be obtained, with probability 1, from a 
single realization X(t,s) of the process. 


A stochastic process can be, in particular, mean ergodic, distribution er- 
godic (that is, ergodic with respect to the distribution function), correlation 
ergodic (with respect to the correlation function), etc. In this book, we will 
limit ourselves to the most important case, namely, the one where the process 
{X(t),t € T} is mean ergodic (see also p. 72). 


Definition 2.3.3. An s.p. {X(t),t € T} for which mx(t) = mVte T is 
called mean ergodic if 


P [ Jim (X(t) = m] =1 (2.26) 
Now, the temporal mean, (X(t))s, is a random variable. Since 


5 Ss 

E|(X(t))s] = os [exe s)ae a ss | mat =m (2.27) 
we can state that Eq. (2.26) in the definition above will be satisfied if the 
variance V[(X(t))s] of the temporal mean decreases to 0 when S tends to 
infinity. Indeed, if lims_,.. V[(X (t))s] = 0, then (X(t))5 converges to its mean 
value when S tends to infinity. To calculate the variance V[(X(t))s], we can 
use the following proposition. 


Proposition 2.3.1. The variance of the temporal mean of the stochastic pro- 


cess {X(t),t € T} is given by 


S S 
VUX)s) = Ze [ ; i: _ Cx (tit) det (2.28) 
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Corollary 2.3.1. If the process {X(t),t € T} is wide-sense stationary, then 
we may write that 


28 8 
vixw@sl= sg ex) [1- EE] as (2.29) 


Often, it is not necessary to calculate V[(X(t))s] to determine whether 
the stochastic process {X(t),t € T} is mean ergodic. For example, when the 
process is WSS, we can use either of the sufficient conditions given in the 
following proposition. 


Proposition 2.3.2. The WSS s.p. {X(t),t € T} ts mean ergodic if 


Cx (0) < co and | lim Cx(s) =0 (2.30) 


s|—00 


or if its autocovariance function Cx (s) is absolutely integrable, that is, if 


a |C'x (s)|ds < 00 (2.31) 


Example 2.3.1. The elementary stochastic process defined in Example 2.2.3 is 
strict-sense stationary. However, it is not mean ergodic. Indeed, we have 


S 
(xy = is | ¥# ay 


Thus, we may write that V[(X(t))s] = V[Y]. Now, if Y is not a constant, the 
variance V[Y] is strictly positive and does not decrease to 0 when S tends to 
infinity (since V[Y] does not depend on S). Therefore, an arbitrary stochastic 
process can be mean ergodic without even being WSS (provided that mx (t) 
is a constant), and a strict-sense stationary process is not necessarily mean 
ergodic. 


Example 2.3.2. As will be seen in Chapter 5, the s.p. {X(t),t © T} called the 
random telegraph signal, which is defined from a Poisson process, is zero mean 
and its autocovariance function is given by 


Cx (s) a e72Als! 


where 4 is a positive constant. Using the conditions in Eq. (2.30), we can state 
that the process is mean ergodic. Indeed, we have 


Cx(0)=1<0oo and -_ Cx(s) = lim en 2Alsl — 9 


8|—00 |s|—00 


Actually, we also have 
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/ iCx(s)ids=2 f eM GSe Se 
oO dr 


—oo 


Remarks. i) It is important to remember that the conditions in Proposition 
2.3.2 are sufficient, but not necessary, conditions. Consequently, if we cannot 
show that the process considered is mean ergodic by making use of this propo- 
sition, then we must calculate the variance of the temporal mean and check 
whether it decreases to 0 or not with S — oo. 


ii) It can be shown that the random telegraph signal is an example of a strict- 


sense stationary stochastic process. Therefore, we can calculate the variance 
V[(X(t))s] by using the formula (2.29): 


V[(X(t))s] = - [ 8 eoDas [1 - 55] 4s 


The integral above is not difficult to evaluate. However, here, it is not even 
necessary to calculate it explicitly. It is sufficient to replace the expression 
between the square brackets by 1, because this expression is comprised of 
values between 0 and 1 when s varies from 0 to 2S. It follows that 


1—e- 48 
VU(X()s] < yg 
Finally, we have 
1— e748 
gim, VIX(O)s] < Jim —Fz— =0 


Since the variance V[(X(t))s] is nonnegative, we can conclude that 
lim V{(X(t))s] =, 
Soo 


which confirms the fact that the random telegraph signal is mean ergodic. 


2.4 Gaussian and Markovian processes 


The bivariate normal distribution was defined in Example 1.3.6. The general- 
ization of this distribution to the n-dimensional case is named the multinormal 
distribution. 


Definition 2.4.1. We say that the random vector (Xj,... ,Xn) has a multi- 
normal distribution if each random variable X;, can be expressed as a linear 
combination of independent random variables Z1,... ,Zm, where Z; ~ N(0,1), 
for j =1,...,m. That is, if 


Xe = bet > 452; fork =1,...,n (2.32) 
j=l 


where pt, 1s a real constant, for all k. 
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Just as a Gaussian distribution N(,¢7) is completely determined by its 
mean j and its variance a”, and a bivariate normal distribution by its pa- 
rameters fix, fly, 0%, o% and p, the joint density function of the random 
vector X = (X,...,Xp) is completely determined by the vector of means 
m := (41x,,--. ,/4x,,) and the covariance matrix K, where 


V[X1] Cov[X1, X92] tame Cov[X1, Xn] 


Cov|X2, X41] V[Xo] ies Cov[X9, Xn] 


Kis (2.33) 


Cov[Xn, X1] Cov[Xn, Xo] see V[Xn] 
By analogy with the one-dimensional case, we write that X ~ N(m,K). 


The matrix K is symmetrical, because Cov[X,Y] = Cov[Y,X], and non- 
negative definite: 


S> So cicgCov[X;, Xk] > 0 Veice ER (2.34) 


i=l k=1 


If, in addition, it is nonsingular, then we may write that 


1 1 1 -1/T T 
= = —-(x —m)K —m 2.35 
Fx(x) (Qr)n/2 (det, K)}/2 exp { 5% ) (x ) ( ) 
for X := (21,...,2n) € R”, where “det” denotes the determinant and < 


denotes the transpose of the vector. 


Proposition 2.4.1. Let X ~ N(m,K). The joint characteristic function of 
the r.v. X: 


Ox(W1,--. Wn) = EB lexp{j(wiXy +... + WnXn)}] (2.36) 


is given by 


ox(w) = exp f So exw - ; > 


i=l 


n 


1 
cas = exp {jan - sku" | 
1 


(2.37) 


where oj, >= Cov[X;, Xp] and w := (wi,... , Wn). 


Proof. We use the fact that any linear combination of Gaussian random 
variables also has a Gaussian distribution. More precisely, we can write that 


Y := uy X1+...+unXn ~ N(py, of) (2.38) 


where 
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Hy (= >= wittx, and oy := So So wiweoir (2.39) 


i=1 i=1 k=1 


We obtain the formula (2.37) by observing (see Table 1.1, p. 19) that 


x (w) = Ele*] = dy(1) = exp (suv - 57) Qo (2.40) 


Properties. i) If Cov[X;, X;| = 0, then the random variables X; and X_ are 
independent. 

ii) If Y;, for i =1,...,m, is a linear combination of the random variables X;, 
of a random vector (X),... ,X,) having a multinormal distribution, then the 
rv. (Y1,...,¥m) also has a multinormal distribution. 


Example 2.4.1. Let X = (X1,...,Xn) be a random vector having a multinor- 
mal distribution N(0,I,,), where 0 := (0,... ,0) and I, is the identity matrix 
of order n. Thus, all the random variables X; have a standard Gaussian dis- 
tribution and are independent (because oj; = 0 V i # j). It follows that the 
mathematical expectation of the square of the distance of the vector X from 
the origin is 


n n 
E[X? + X2+...+X2}= > ° B[X}] "2 Diagn 
k=1 k=1 


and the variance of the squared distance is 


n n 
VIX? +X34...4.X2] 2S ViXz]'S S72 = 20 
k=1 k=1 


because if Z ~ N(0,1), then we have 
4 ce /2 
ae Pee a —~—W = 
El4 a ( j) duy4 : a : 


so that V[Z?] = 3 — 12 =2. 


Definition 2.4.2. A stochastic process {X(t),t € T} is said to be a Gaus- 
sian process if the random vector (X(t1),...,X(tn)) has a multinormal dis- 
tribution, for any n and for allt),... ,tn.- 


Remark. Let X be a random variable whose distribution is N(j x02): Any 
affine transformation of X also has a Gaussian distribution: 


Y:=aX+b = YN(aux +6,070%) (2.41) 
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Similarly, any affine transformation of a Gaussian process remains a Gaussian 
process. For example, if {X(t),t € T} is a Gaussian process, then the s.p. 
{Y (t),t € T} defined by 


Y(t) =2X(t)-1 or Y(t) = X(t?) (2.42) 
is Gaussian as well. We can also show that {Y(t),¢ € 7} is a Gaussian process 
if 

t 
Y(t) = / X(s) ds (2.43) 
0 


because an integral is the limit of a sum. However, the process is not Gaussian 
if 
Y(t)= X(t) or Y(t) =e*© (2.44) 


etc. 


Proposition 2.4.2. If a Gaussian process {X(t),t € T} is such that its mean 
mx (t) ts a constant mx and if its autocovariance function Cx(t,t+s) depends 
only on s, then it ts stationary (in the strict sense). 


Proof. Since the nth-order characteristic function of the s.p. {X (t),t € T} is 
given by {see Eq. (2.37)] 


E b {idee} = = exp {ime 3 So wi - Ly oxt itr wan 
i=1 i=l k=1 
(2.45) 


we can assert that the statistical characteristics of a Gaussian process de- 
pend only on its mean and its autocovariance function. Now, we see that the 
function above is znvariant under any change of time origin. 0 


Remark. The preceding proposition means that a wide-sense stationary Gaus- 
sian process is also strict-sense stationary. 


Definition 2.4.3. An s.p. {X(t),t € T} is said to be Markovian if 
P[X (tn) < Zn | X(t),t S tra] = P[X (tn) Son | X(tn-1)] (2.46) 
where tn. < tn. 


Remarks. i) We say that the future, given the present and the past, depends 
only on the present. 


ii) If {X(t),t € T} is a discrete-state process, we can write the formula (2.46) 
as follows: 
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P[Xt+s =j | Xt a 1, Xt,, =tny--- Xt — i] am P[Xe+s =j | Xt oa j] (2.47) 
for all states i),...,in,2,j, and for all time instants t1 < ... < tn < ¢ and 


s>0. 


iii) The random walk considered in Example 2.1.1 is a typical example of a 
Markovian process, which follows directly from the fact that we assume that 
the tosses of the coin are independent. 


A Markovian, continuous-time, and continuous-state stochastic process, 
{X(t),t € T}, is completely determined by its first-order density function: 


) 
f(z;t) = —P[X(t) < z] (2.48) 
Ox 
and by its conditional transition density function, defined by 
p(z, £0; t, to) = fx (e)|x(to)(z | Zo) (2.49) 
ig ES ee eae | Xo) SO pg a 
dx} dx 


Since the process must be somewhere at time t, we have 


oO oo 
i f(z;t)dz =1 and i p(x, 2o;t, to) dx =1 (2.50) 
—0o —90 


Moreover, by conditioning on all possible initial states, we may write that 
oO 
slat) = [ f(x0;to)p(a,205tsto) da (2.51) 
—o0o 


Similarly, we deduce from the Chapman*—Kolmogorov’ equations (see 
Chapter 3) that 


OO 


ple, o;t,to) = f p(x, £1; t, t1)p(r1, £0; t1, to) dry (2.52) 


—oO 


where to < t; < t. Finally, since at the initial time the distribution of the 
process is deterministic, we also have 


4 Sydney Chapman, 1888-1970, was born in England and died in the United States. 
He is especially known for his work in geophysics. One of the craters of the moon 
is named after him. 

° Andrei Nikolaevich Kolmogorov, 1903-1987, was born and died in Russia. He was 
a great mathematician who, before getting his Ph.D., had already published 18 
scientific papers, many of which were written during his undergraduate studies. 
His work on Markov processes in continuous time and with continuous-state space 
is the basis of the theory of diffusion processes. His book on theoretical probability, 
published in 1933, marks the beginning of modern probability theory. He also 
contributed in an important way to many other domains of mathematics, notably 
to the theory of dynamical systems. 
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lim p(z, Xo; t, to) = d(x — xo) (2.53) 
Oo 


where 6(-) is the Dirac® delta function defined by 


d(x) = {e i # ; (2.54) 
so that 
a d(x)dx =1 (2.55) 


Definition 2.4.4. The infinitesimal mean m(z;t) and the infinitesimal 
variance vu(x;t) of the continuous-time and continuous-state stochastic pro- 
cess {X(t),t € T} are defined, respectively, by 
1 
teh) = lim ElX(t +e) —X(t)| X(t) =2] (2.56) 
€ 
and 


u(x;t) = ae ~E((X(t +) — X(t))? | X(t) =a] (2.57) 


Remarks. i) We can also obtain m(x9; to) and v(x9; to) as follows: 


nen) =i EKG) EO Sal (2.58) 
t\to Ot 
and 
v(¢os to) = lim VIX (8) | X(to) = xo) (2.59) 


ii) Suppose that the process {X(t),¢ € T} has infinitesimal moments m(z; t) 
= m/(x) Vt and v(x; t) = v(x) and that its state space is the interval [a, 5] (or 
[a, b), etc.). Let 


Y(t) :=g{X(t)] forteT (2.60) 


If the function g is strictly increasing or decreasing on the interval [a,b] and 
if the second derivative g(x) exists and is continuous, for a < x < b, then 


® Paul Adrien Maurice Dirac, 1902-1984, was born in England and died in the 
United States. He was a physicist whose father was a French-speaking Swiss. He 
won the Nobel Prize for physics in 1933 for his work on quantum mechanics. He 
was a professor at the University of Cambridge for 37 years. 
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we can show that the infinitesimal moments of the process {Y(t),t € T} are 
given by 


my (y) = m(x)g'(x) + 5v(2)9"(e) (2.61) 
and 


vy(y) = v(a)[g'(@))? (2.62) 
where z = g~*(y). Moreover, the state space of the process is the interval 
(g(a), g(b)} (respectively, [9(b), g(a)]}) if g is strictly increasing (resp., decreas- 
ing). 

It can be shown that the function p(x, x09;t, to) satisfies the following par- 
tial differential equations: 


1 


Op oO 1 2? 
: oe ee . _ 2.63 
Se + [m(x3t) p] — Salut) pl = 0 (2.63) 
and 
Op Op i a*p 
. agree — id —_—_-—= 2. 
Bt + m(zto; to) 5 + 5 0(a05 to) Ba? 0 (2.64) 


These equations are called the Kolmogorov equations or the diffusion equa- 
tions. The first one is the Kolmogorov forward equation (or Fokker’ -Planck® 
equation), and the second one is the Kolmogorov backward equation. 


Definition 2.4.5. If the function p(z,z0;t,to) depends only on (x, xo and) 
the difference t — to, the stochastic process {X(t),t € T} is said to be time- 
homogeneous. 


Remarks. i) If the s.p. {X(t),t € T} is time-homogeneous, then the functions 
m(z;t) and v(x; t) do not depend on t. 

ii) Note that a time-homogeneous s.p. is not necessarily (wide-sense) station- 
ary, because the function f(z;t) may depend on the variable t. On the other 
hand, if f(z;t) = f(x), then the process is strict-sense stationary, because it 
is completely determined by f(x;t) and p(z, o3t, to). 


Definition 2.4.6. A stochastic process {B(t),t > 0} is called a white noise 
(process) if its mean is equal to zero and if its autocovariance function is of 
the form 


Ca(t1, ta) = g(t1) d(t2 — t1) (2.65) 
where q(t1) > 0 and 6(-) is the Dirac delta function (see p. 63). 
” Adriaan Daniél Fokker, 1887-1972, was born in Indonesia and died in the Nether- 
lands. He was a physicist and musician. He proved the equation in question in 
1913 in his Ph.D. thesis. 


8 Max Karl Ernst Ludwig Planck, 1858-1947, born and died in Germany, was a 
renowned physicist famous for the development of quantum theory. 
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Remark. Actually, {B(t),t > 0} is not a stochastic process in the proper sense 
of the term. 


Finally, the following important result can be shown. 


Proposition 2.4.3. A Gaussian and stationary process, {X(t),t € T}, 4 
Markovian if and only if its autocorrelation function is of the form 


Rx(s) = o7e7 “5! (2.66) 
where @ and a are positive constants. 


Example 2.4.2. The Wiener process {W(t),t > 0} (see Example 2.2.1) is such 
that E[W(t) | W(to) = wo] = wo and V[W(t) | W(to) = wo] = o7(t — to). 
It follows that its infinitesimal mean and variance are given by m(w;t) = 
0 and v(w;t) = o?. Therefore, the conditional transition density function 
p(w, wo; t,to) satisfies the Kolmogorov forward equation 


Op 18 
Bt 2Be2? 2) = 9 


We can check that W(t) | {W(to) = wo} has a Gaussian distribution with 
parameters wo and o?(t — to). That is, the function p(w, wo; t, to) given by 


1 (w — wo)? 


a exp ( - =? fort > ty 
2ra*(t — to) pf a 


p(w, wo;t, to) = 


is a solution of the partial differential equation above. Moreover, we have 


lim p(w, wo; t, to) = é(w _ wo) 
tlto 


as required. 
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Section 2.1 


Question no. 1 
We define the stochastic process {X(t),t > 0} by 


X(t)=e"¥* fort>0 
where Y is a random variable having a uniform distribution on the interval 
(0,1). Calculate 
(a) the first-order density function of the process {X(t),t > 0}, 
(b) E[X(é)], for t > 0, 
(c) Cy (t,t + 8), where s,t > 0. 
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Question no. 2 

Let {X,,n = 1,2,...} be a Bernoulli process. That is, the random vari- 
ables X,,X9,... are independent and all have a Bernoulli distribution with 
parameter p. Calculate 


(a) the particular case p(0,1;n1 = 0,n2 = 1) of the second-order probability 
mass function of the process, 


(b) the correlation coefficient px(n,m) of the process if p = 1/2 and n,m € 
‘ca ae 


Question no. 3 
Calculate the first-order density function of the s.p. {X(t),t => 0} defined 
by 
X(t)=tY +1 


where Y is a random variable having a U(0,1) distribution. 


Question no. 4 
We define the stochastic process {X(t),t > 0} by 


X()= = for t > 0 
where Y has a uniform distribution on the interval (0,2). Calculate the func- 
tion f(x;t), for x > t/2. 


Question no. 5 

We consider the process {X(t),t > 0} defined by X(t) = (tanY)t, for 
t > 0, where Y is a random variable having a uniform distribution on the 
interval (—71/2, 7/2). Calculate the probability P[5 t € (0,1): X(t) ¢ [0,1]. 
In other words, calculate the probability that the process {X(t),t > 0} leaves 
the interval [0,1] between 0 and 1. 


Indication. It can be shown that the r.v. W := tan Y has the following density 


function: : 
=——~—— _ fi eR 
fw(w) m(w? + 1) or w 
That is, W has a Cauchy? distribution, or a Student! distribution with one 
degree of freedom. 


® Augustin Louis Cauchy, 1789-1857, was born and died in France. He is considered 
the father of mathematical analysis and the inventor of the theory of functions 
of a complex variable. 

10 Pseudonym of William Sealy Gosset, 1876-1937, who was born and died in En- 
gland. He worked as a chemist for the Guinness brewery, in Ireland, where he 
invented a statistical test for the control of the quality of beer. This test uses the 
distribution that bears his name. 
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Question no. 6 
Are the increments of the stochastic process {X(t),t > 0} defined in Ex- 
ample 2.1.6 independent? stationary? Justify. 


Question no. 7 

(a) Find the autocorrelation function of the process {X(t),¢ > 0} defined by 
X(t)=1ifY >t and X(t) =O if Y <¢, where Y is a random variable having 
a U(0,c) distribution. 

(b) Calculate Rx (t1,t2) in (a) if X(t) = 6(Y — t) instead, where 4(-) is the 
Dirac delta function. 


Question no. 8 

We consider the process {X(t),t > 0} defined by X(t) = e~**, for t > 0, 
where Y is a continuous random variable whose density function is fy (y), for 
y 2 0. 


(a) Find f(a;¢) in terms of fy(y). 


(b) Calculate E[X(t)] and Rx(t1,t2) when Y has an exponential distribution 
with parameter 1. 


x= f Lf Bevar| ds fort >0 


where {B(t),t > 0} is the white noise process defined on p. 64. What is the 
average power of the stochastic process {X(t),¢ > 0}? 


Question no. 9 
Let 


Section 2.2 


Question no. 10 
Is the stochastic process {X(t),t > 0} defined in Question no. 1 wide-sense 
stationary? Justify. 


Question no. 11 
Let {X(t),¢ > 0} be a stochastic process whose autocorrelation and auto- 
covariance functions are 


Rx(t1,t2) = ev ttm tal +1 and Cx (ti, te) = ew ltt 


Is the process wide-sense stationary? Justify. 


Question no. 12 
Let {X(t),t > 0} be a wide-sense stationary stochastic process, with zero 
mean and autocorrelation function given by Rx(s) = e7!*!. We define 


Y(t) =tX?(1/t) fort >0 


Is the stochastic process {Y (t),t > 0} wide-sense stationary? Justify. 
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Question no. 13 
We consider a stochastic process {X(t),t > 0} for which 


1 1 
Cx(ty, te) = ——__—— and Rx(t,t.)=——_—_—_— +4 
x (t1, t2) oh td an x (ti, t2) eee 
Is the process wide-sense stationary? Justify. 


Question no. 14 
Let Y be a random variable having a uniform distribution on the interval 
(—1,1). We consider the stochastic process {X(t),t > 0} defined by 


X(t)=Y*%t fort >0 


Is the process stationary? J ustify. 


Question no. 15 
Let Y be a random variable such that dy(1) = 2¢y(—1) and ¢y(2) = 
4g@y (—2), where dy(-) is the characteristic function of Y. We set 


X(t) =cos(wt + Y) fort >0 


Show that the stochastic process {X(t),t > 0} is wide-sense stationary. 
Indication. We have the following trigonometric identity: 


cos(wt, + s) cos(wt2 +s) = ; {cosw(t — ta) + cos(wt, + wt, + 2s)} 


Question no. 16 
We set 
Y(t)= X(t+1)-—X(t) forteR 
where {X(t),t € R} is a wide-sense stationary stochastic process. Find the 


spectral density of the process {Y(t),t € R} in terms of Sx (w). 


Question no. 17 
Consider the wide-sense stationary process {X(t),t € R} whose spectral 
density is Sx(w) = 2/(1+w?). We set 
t41 
Y(t) = X(s)ds forteR 


t-1 


Calculate Sy (w). 
Indication. Write Y(t) in the form 


Y(t) = 7 h(t — s)X(s) ds 


for an appropriately chosen function h. 
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Question no. 18 
We consider the random variable Y(t) defined by Eq. (2.19). Calculate the 
mean of X(t)Y(t) if {X(t),t € T} is a white noise process. 


Section 2.3 


Question no. 19 
Are the stochastic processes defined in Questions nos. 11, 12, and 13 mean 
ergodic? Justify. 


Question no. 20 


Is the Bernoulli process defined in Example 2.1.5 mean ergodic? Justify. 


Question no. 21 
We define X(t) = m+B(t), for t > 0, where m is a constant and {B(t),t > 
0} is a white noise process, so that E[B(t)| = 0 and 


Cp(ti, te) = q(ti) 6(te — t1) 


where 6(-) is the Dirac delta function. Show that the process {X(t),t > 0} is 
mean ergodic if the function g is bounded. 


Question no, 22 
Let {X(t),¢ > 0} be a wide-sense stationary stochastic process, with zero 
mean and for which 
Rx(s) = e7 “5! cos 2aws 


where a and w are positive constants. Is this process mean ergodic? Justify. 


Question no. 23 
The stochastic process {X (t),t > 0} is defined by 


X(t) =Y + Bt) 


where Y ~ N(0,1) and {B(t),t > 0} is a white noise process whose autocor- 
relation function is Rg(s) = cd(s), with c > 0. We assume that the random 
variable Y and the white noise are independent. Is the process {X(t),t > 0} 


(a) wide-sense stationary? 
(b) mean ergodic? 
Question no. 24 


Let {B(t),t > 0} be a white noise process for which the function q(t1) is 
a constant c > 0. Calculate the variance of the random variable 


1 T 
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Question no. 25 

The standard Brownian motion is the particular case of the Wiener process 
{W(t),t > 0} (see Example 2.2.1) for which E(W(t)] = 0 and Cw(ti,t2) = 
min{ty, to}. 
(a) Calculate V[(W(t)) 5]. 
(b) Is the process mean ergodic? Justify. 


Section 2.4 


Question no. 26 

Let (Xy,...,X,) be a random vector having a multinormal distribution 
for which m = (0,... ,0) and the covariance matrix K is the identity matrix 
of order n (> 4). 
(a) Calculate the characteristic function of Y := Xy + Xo. 
Reminder. The characteristic function of a random variable X having a Gaus- 


sian N(u, 07) distribution is given by ¢x(w) = exp (juw — 307w?). 


(b) Does the random vector (Y,Z), where Z := X3 — X4, have a bivariate 
normal distribution? If it does, give its five parameters; otherwise, justify. 


Question no. 27 
Let {X(t),t > 0} be a Gaussian process such that E[X(t)] = pt, for t > 0, 
where jz is a nonzero constant, and 


Rx(t,t+s) =2t+p7t(t+s) for s,t>0 
We define Y(t) = X(t) — ut, for t > 0. Is the process {Y(t),t > 0} stationary? 
Justify. 


Question no. 28 
We consider a Gaussian process {X(t),t > 0} for which E[|X(t)] = 0 and 
whose autocovariance function is 


Cx(t,t+s)=e7' for s,t>0 
Let Y(t) := X(t), for t > 0. Is the stochastic process {Y(t),t > 0} (a) sta- 
tionary? (b) mean ergodic? Justify. 


Question no. 29 

Let {X,,n = 1,2,...} be a discrete-time and discrete-state process such 
that 
PiXayi = 5 | Xn = 1, Xn-1 = tn-i,... ,X1 = th] 


= P[Xn41 = 7 | Xn =41, Xn-1 ae tcl 


for all states 1),... ,%,-1,1,7 and for any time instant n. This process is not 
Markovian. Transform the state space so that the process thus obtained is 
Markovian. 
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Question no. 30 
Suppose that {X(t),t > 0} is a Gaussian process such that 


E(|X(t) | X (to) = Xo] =o and V[X(t) | X (to) = Xo] =t Vt >0 
Let 
Y(t):=e* fort >0 


(a) Calculate the infinitesimal parameters of the process {Y(t),¢ > 0} by 
using the formulas (2.56) and (2.57). 


(b) Check your results with the help of the formulas (2.58) and (2.59). 


Question no. 31 
Find a solution of the Fokker-Planck equation 


Op O 1.6? 
vans 3, HP) = 3 9n2 (7 


where pz € R and o > 0 are constants, for which 


2») =0 (for x € R and t > ty > 0) 


lim p(x, x9; t, to) = 6(x — Xo) 
tlto 


Indications. i) Try the density function of a Gaussian distribution as a solution. 
ii) We can take the Fourier transform (with respect to x) of the equation and 
then invert the solution of the ordinary differential equation obtained. We 
have 
lim p(x, 29;t,t)) = 0 
Hi ge) 


which implies that 


LLCO 


lim Ora caeebeias =0 
Ox 
as well. 


Question no. 32 

Let {X(t),¢ € R} be a wide-sense stationary Gaussian process, with zero 
mean and Rx(s) = 2e~4l*!, We define Y = X(t+s) and Z = X(t—s), where 
s>0. 


(a) What is the distribution of the random variable X(t)? 
(b) Calculate the mean of YZ. 
(c) Calculate the variance of Y + Z. 


Question no. 33 
The Gaussian process {X (t),t > 0} is such that E[X(t)] =0 Vt and 


Rx (t,t) = ee 


where cj, C2 > 0. 
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(a) Show that the stochastic process {X(t),t > 0} is stationary and mean 
ergodic. 


(b) We define the process {Y (t),t > 0} by 


1 if X(t) < 
A i if X(t) >a 


We then have E[Y(t)] = F(a;t). If the process {X(t),¢ > 0} is stationary, 
we may write that F(z;t) = F(x). We say that the s.p. {X(t),t > O} is 
distribution ergodic if {Y(t),t > 0} is mean ergodic. Is the stochastic process 
{X(t),t > 0} distribution ergodic? 


(c) When the process {X(t¢),t > 0} is stationary, we also define the process 
{Z,(t),t > 0} by Z(t) = X(t)X(t+s), where s > 0. It follows that E[Z,(t)] = 
Rx(s). We say that the s.p. {X (t),t > 0} is correlation ergodic if {Z,(t),t > 0} 
is mean ergodic. Can we assert that {X(t),t > 0} is correlation ergodic? 


Question no. 34 
Suppose that {X(t),t € R} is a stationary Gaussian process such that 


Cx(s) = Rx(s) =e?! forse R 


(a) What is the distribution of X(t)? Justify. 
(b) We define the random variable 


Y(t) = BX(t)] +1 
where &(-) is the distribution function of the N(0,1) distribution. That. is, 
(z) := PIN(0,1) < z]. What is the distribution of Y(t)? Justify. 
Indication. The inverse function S~} exists. 


Question no. 35 
Let {X(t),¢ € R} be a stationary Gaussian process whose autocovariance 


function is [1 — (|s1/2)] if Jt 
All — ({s|/2)] if |s| < 2 
Cx(e) = { 0 if |s| >2 
Is the stochastic process {X(t),¢ € R} 
(a) mean ergodic? 
(b) distribution ergodic (see Question no. 33)? Justify. 


3 


Markov Chains 


3.1 Introduction 


The notion of a Markovian process was seen in Section 2.4. In the general 
case, the stochastic process {X(t),t € T} is said to be Markovian if 


P(X (tn41) EA | X(t) =2M,t< tn| = P(X (tn41) EA | X (tn) = Lt, (3.1) 


for all events A and for all time instants ty < tn4i- 


Equation (3.1) means that the probability that the process moves from 
state x,,, where it is at time ¢,, to a state included in A at time t,4; does not 
depend on the way the process reached x;, from x;,, where tg is the initial 
time (that is, does not depend on the path followed by the process from <;, 
to x,,,). 

In this chapter, we will consider the cases where {X(t),t € T'} is a discrete- 
time process and where it is a continuous-time and discrete-state process. 
Actually, we will only mention briefly, within the framework of an example, 
the case of discrete-time and continuous-state processes. 

When {X,,n = 0,1,...} is a discrete-time and discrete-state process, the 
Markov property implies that 


P[Xn41 = 9 | Xp =1,Xn-1 =tn-1,--- ,Xo = to) = P[Xn41 =j| Xn =4) 


(3.2) 
for all states t9,... ,in-1,1,7, and for any time n. We also have 
PiXn41 = j | Xn-1 = tn—1, Xn-2 = tn-2)--- ,Xo = io] 
= P[Xn4i =j Xn-1 = tn—1] (3.3) 


etc., which means that the transition probabilities depend only on the most 
recent information about the process that is available. 
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Remark. We call a Markov chain a discrete-time process that possesses 
the Markov property. Originally, this expression denoted discrete-time and 
discrete-state Markovian processes. However, we can accept the fact that 
the state space of the process may be continuous. By extension, we will call 
a continuous-time Markov chain a discrete-state (and continuous-time) 
Markovian stochastic process. Discrete-time and continuous-time Markov 
chains will be studied in Sections 3.2 and 3.3, respectively. 


We can generalize Eq. (3.2) as follows. 


Proposition 3.1.1. If the discrete-time and discrete-state stochastic process 
{Xn,n =0,1,...} possesses the Markov property, then 


Fa ©. es ee, ee ee .) EB | AnH ins Ane = lasses: ,X0 = io] 
= Pl(Xn41; Xn42) Ar .) EB | Xn = in| (3.4) 


where B is an infinite-dimensional event. 


Remark. For example, by using the fact that 
P[ANBNC) — P[ANBNC] PIBNC) 
P{C} ~ PIBNC] Pic 
= P|A| BNC)P[B|C] (3.5) 


P[ANB|C]= 


we can easily show that 


Pl{(Xni1, Xn42) = (in+1, in42) | Xn oe ins Xn-1 = in-1) sae , Xo = io] 
mad Pl(Xn41, Xn42) = (ins1;tn42) | Xn = tn] (3.6) 


We now give several examples of discrete-time and continuous-time Markov 
chains. 


Example 3.1.1. One of the simplest, but not trivial, examples of a Markovian 
process is that of the random walk (see p. 48). We can generalize the random 
walk as follows: if the particle is in state 7 at time n, then it will be in state 
j at time n + 1 with probability 


pifg=it+l 
— _4_ )Jq@ifg=i-l 
PI] Me ie ag 


0 otherwise 


where p, q, and r are nonnegative constants such that p+ q+r = 1. That 
is, we add the possibility that the particle does not move from the position it 
occupies at time n. 

Another way of generalizing the random walk is to assume that the prob- 
abilities p and g (and r in the case above) may depend on the state 7 where 
the particle is. Thus, we have 
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Pi if 7 =i2+1 
P[Xng1 =F | Xn == q@ ifj=t-1 (3.7) 
0 otherwise 


and p; + q@ = 1, for any i € {0,41,+2,...}. 

Finally, we can generalize the random walk to the d-dimensional case, 
where d € N. For example, a two-dimensional (d = 2) random walk is a 
stochastic process {(X;,, Y,),n =0,1,...} for which the state space is the set 


S(x,.,¥n) = {(i, 7): $29 E {0, +1, +2, tars yh 
and such that 
PIX Yak) = (dis ieaa) | (Xn Yn) = Garda) 


py if int. = tn +1, Jn41 = Jn 
P2 if Int = ins Jn+1 _ Jn +1 
= q if In+1 = tn a 1, jn+1 = Jn 

q2 if nga = tn; jnt1 = In — 1 

0 otherwise 
where p; + po +41 +q2 = 1 (and p; > 0 and q; > 0, for 7 = 1,2). Thus, the 
particle can only move to one of its four nearest neighbors. In three dimensions, 
the particle can move from a given state to one of its six nearest neighbors [the 
neighbors of the origin being the triplets (+1,0,0), (0, +1,0) and (0,0, +1)], 
etc. 


Example 3.1.2. An important particular case of the random walk having the 
transition probabilities given in Eq. (3.7) is the one where 


N-1 z 
Pi = N and qi = WN 
If we suppose that the state space is the finite set {0,1,...,N}, then we can 


give the following interpretation to this random walk: we consider two urns 
that contain N balls in all. At each time unit, a ball is taken at random from 
all the balls and is placed in the other urn. Let X, be the number of balls in 
urn I after n shifts. This model was used by Paul and Tatiana Ehrenfest’ to 
study the transfer of heat between the molecules in a gas. 


Example 3.1.3. Suppose that we observe the number of customers that are 
standing in line in front of an automated teller machine. Let X,, be the number 
of customers in the queue at time n € {0,1,...}. That is, we observe the 
system at deterministic time instants separated by one time unit, for example, 


1 Paul Ehrenfest, 1880-1933, born in Austria and died in the Netherlands, was 
a physicist who studied statistical mechanics and quantum theory. His wife, of 
Russian origin, was Tatiana Ehrenfest-Afanaseva, 1876-1964. She was interested 
in the same subjects. 
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every hour, or every 10 minutes (a time unit may be an arbitrary number of 
minutes), etc. Suppose next that if a customer is using the teller machine at 
time n, then the probability that he is finished before time n + 1 is g € (0,1). 
Finally, suppose that the probability that k customers arrive during a time 
unit is given by 
_ 
P[¥Yn = k] =e a 
for k = 0,1,... and for any n € {0,1,...}. That is, the number Y,, of arrivals 
in the system in the interval [n,n +1) has a Poisson distribution with parame- 
ter A Vn. Thus, we may write that Xn41 = Xn+Yp, with probability p = 1—q 
and Xn4i1 = (Xn—1)+Y, with probability g. We find that {X,,n =0,1,...} 
is a (discrete-time and discrete-state) Markov chain for which the transition 
probabilities are 


Be age rere 
pe Et cep een 
P{Xn41=j|Xn =i] = ae if 7 =i-landieN 
ey if j7 =k andi=0 
0 otherwise 


(3.8) 
This is an example of a queueing model in discrete-time. 


Example 3.1.4. An example of a discrete-time, but continuous-state, Markov 
chain is obtained from a problem in optimal control: suppose that an object 
moves on the real line. Let X, be its position at time n € {0,1,...}. We 
suppose that 

Xn41 = AnXn + bn Un + En 


where @,, and b, are nonzero constants, uy, is the control (or command) vari- 
able, and €, is the term that corresponds to the noise in the system. We 
assume that e, ~ N(0,o7), for all n and that the random variables X, and 
€n are independent. The objective is to bring the object close to the origin. 
To do so, we choose 


so Xn Vn (3.9) 


Un = — 
Remark. In stochastic optimal control, the variable un, must be chosen so as 
to minimize the mathematical expectation of a certain cost function J. If the 
function in question is of the form 


oO 


n=0 


where c > 0 and k are constants, then the optimal solution is indeed obtained 
by using the formula (3.9) above. 
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The s.p. {Xn,n =0,1,...} is a Markov chain. If fx,,,,1x,, (y | Z) denotes 
the conditional density function of the random variable X41, given that 
Xp = XL, we may write that 


1 y? 
= ——-e -~— f ER 
PXniilXn (y | z) J/2no x{ o } or y 
That is, Xni | Xn ~ N(0,07). 
We can also suppose that the noise €, has a variance depending on Xy. For 
instance, V[e,] = o?X2. In this case, we have that Xn41 | Xn ~ N(0,07X?2). 


Example 3.1.5. Suppose that X(t) denotes the number of customers who ar- 
rived in the interval [0, t] in the example of a queueing system in discrete-time 
(see Example 3.1.3). If we assume that the time 7, elapsed until the arrival 
of a new customer into the system, when there have been n arrivals since 
the initial time, has an exponential distribution with parameter A, for all n, 
and that the random variables 79, 7,,... are independent, then we find that 
{X(t),t > 0} is a continuous-time and discrete-state Markovian process called 
a Poisson process. In this case, the number of arrivals in the system in the 
interval [0,t} has a Poisson distribution with parameter At. Therefore, during 
one time unit, the number of arrivals indeed has a Poi(A) distribution. 

In general, if the random variable 7, has an exponential distribution with 
parameter Ax, then {X(t),t > 0} is a continuous-time Markov chain. 


Example 3.1.6. In the preceding example, we considered only the arrivals in 
the system. Suppose that the time an arbitrary customer spends using the 
automated teller machine is an exponential random variable with parameter 
p. Now, let X(t) be the number of customers in the system at time t > 0. 
If the customers arrive one at a time, then the process {X(t),t > 0} is an 
example of a continuous-time Markov chain called a birth and death process. It 
is also the basic model in the theory of queues, as will be seen in Chapter 6. It 
is denoted by the symbol M/M/1. When there are two servers (for example, 
two automated teller machines) rather than a single one, we write M/M/2, 
etc. 


Remark. Actually, the Poisson process is a particular case of the so-called pure 
birth (or growth) processes. 


3.2 Discrete-time Markov chains 


3.2.1 Definitions and notations 


Definition 3.2.1. A stochastic process {Xn,n = 0,1,...} whose state space 
Sx,, is finite or countably infinite is a stationary (or time-homogeneous ) 
Markov chain if 
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P[Xaei =Jj | Xn = i, Xn-1 = tn-1)--- Xo = io| 
= P[Xn41 =j| Xn =i] = Diy (3.10) 


for all states i9,... ,in-1,1,7 and for any n > 0. 


Remarks. i) In the general case, we can denote the conditional probability 
P(Xn4i = 9 | Xn = i] by p;,;(n). Moreover, when there is no risk of confusion, 
we may also write p;,; simply as p;;. 

ii) When the states of the Markov chain are identified by a coding system, we 
will use, by convention, the set N° := {0,1,... } as the set space. For example, 
suppose that we wish to model the flow of a river as a Markov chain and that 
we use three adjectives to describe the flow X, during the nth day of the year: 
low, average, or high, rather than considering the exact flow. In this case, we 
would denote the state low flow by state 0, the state average flow by 1, and 
the state high flow by 2. 


iii) Note that, in the example given above, the conditional probabilities 
P[Xn41 = J | Xn = i] actually depend on n, since the probability of moving 
from a low flow to an average flow, in particular, is not the same during the 
whole year. Indeed, this probability is assuredly smaller during the winter and 
higher in the spring. Therefore, we should, theoretically, use a nonstationary 
Markov chain. In practice, we can use a stationary Markov chain, but for a 
shorter time period, such as the one covering only the thawing period in the 
spring. 

In this book, we will consider only stationary Markov chains. Anyhow, 
the general case is not used much in practice. Moreover, in the absence of an 
indication to the contrary, we take for granted in the formulas and definitions 
that follow that the state space of the chain is the set {0,1,...}. 


Definition 3.2.2. The one-step transition probability matrix P of a 
Markov chain is given by 


0 1 2 
0} Po,o Po,1 Po,2 --- 
1] P1,0 P1,1 P1,2 --- 


= 21 P20 P21 P2,2 --- (3.11) 


Remarks. i) We have indicated the possible states of the Markov chain to the 
left of and above the matrix, in order to facilitate the comprehension of this 
transition matrix. The state to the left is the one in which the process is at 
time n, and the state above that in which the process will be at time n + 1. 


ii) Since the p;,;’s are (conditional) probabilities, we have 


Pig 20 Vig (3.12) 
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Moreover, because the process must be in one and only one state at time n+1, 
we may write that 


oO 
Sopig=1 Vi (3.13) 
j=0 


A matrix that possesses these two properties is said to be stochastic. The sum 
20 Pi,js for its part, may take any nonnegative value. If we also have 


oO 
So pig =1 V5 (3.14) 
i=0 


the matrix P is called doubly stochastic. We will see, in Subsection 3.2.3, that 
for such a matrix the limiting probabilities, in the case when the state space 
is finite, are obtained without our having to do any calculations. 


We now wish to generalize the transition matrix P by considering the case 
when the process moves from state i to state j in n steps. We then introduce 
the following notation. 

Notation. The probability of moving from state i to state 7 in n steps (or 
transitions) is denoted by 


Bh = PiXmin =J|Xm =i], for m,n,i,7 20 (3.15) 


From the pa we can construct the matrix P of the transition prob- 
abilities in n steps. This matrix and P have the same dimensions. Moreover, 
we find that we can obtain P™ by raising the transition matrix P to the 


power n. 


Proposition 3.2.1 (Chapman—Kolmogorov equations). We have 
lee) 
pet”) = Sp") for m,n,i,j > 0 (3.16) 
k=0 


Proof. Since we consider only stationary Markov chains, we can write, using 
the total probability rule, that 


Be”) = PiXmin =5| Xo = 4] (3.17) 
CO 

=) Pita Soe =F Xo 4 (3.18) 
k=0 


We then deduce from the formula (see p. 74) 


P[ANB|C]=P[A| BnCjP[B|C| (3.19) 
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that 


(m+n) _ 
iJ _ 


Je 


P\Xman =j | Xm =k, Xo =iP[Xm =k|Xo= 7 


co 
il 
So 


i 


foe) 
= De Dy py) oO (3.20) 
k=0 


rod 
HT 


0 


In matricial form, the various equations (3.16) are written as follows: 


pim+n) — p(m) p(n) (3.21) 
which implies that 
P® = pM p...p@ (3.22) 
ne” 
mx 


Since P™ = P, we indeed have 


p() pn (3.23) 


Example 3.2.1. Suppose that the Markov chain {X,,n = 0,1,...}, whose 
state space is the set {0,1, 2}, has the (one-step) transition matrix 


0 1 2 
01/3 1/3 1/3 
P=1] 0 1/21/2 
2} 1 0 0 
We have 
4/9 5/18 5/18 
Pp?) = p? = | 1/2 1/4 1/4 
1/3 1/3 1/3 


Thus, pee = 4/9. Note that to obtain this result, it is sufficient to know the 


first row and the first column of the matrix P. Similarly, if we are looking for 
ph), we can use the matrix P() to calculate the first row of P“). We obtain 


139/324 185/648 185/648 
p) = p4 = ee = 


Next, we have 


) 139. 1 185 185 833 
ei eas eee sao BoA RB 
Poo = 354 ~ 3 + Gag ~°* Gag *! = Toad 
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So, in general, to obtain only the element ph” of the matrix P™, it suffices 
to calculate the ith row of the matrix P(~) and to multiply it by the jth 
column of P. 

In Subsection 3.2.3, we will be interested in finding (if it exists) the limit 
of the matrix P() when n tends to infinity. To do so, we will give a theorem 
that enables us to calculate the limiting probabilities of the Markov chain. 
By making use of a mathematical software package, we can also multiply 
the matrix P by itself a sufficient number of times to see to what matrix it 
converges. Here, we find that if n is sufficiently large, then 


0.4286 0.2857 0.2857 
P™) — p” ~ | 0.4286 0.2857 0.2857 
0.4286 0.2857 0.2857 


Observe that the three rows of the matrix above are identical, from which 
we deduce that whatever the state in which the process is at the initial time, 
there is a probability of approximately 0.4286 that it will be in state 0 after a 
very large number of transitions (this probability was equal to ~ 0.4285 after 
only five transitions, from state 0). By using the theorem of Subsection 3.2.3, 
we can show that the exact probability that the process is in state 0 when it 
is in equilibrium is equal to 3/7. 


Sometimes, rather than being interested in the transition probabilities in 
n steps of the Markov chain {X,,n = 0,1,...}, we want to calculate the 
probability that the chain will move to state j, from Xo = 3, for the first time 
at time n. 
Notation. The probability of moving to state 7, from the initial state 7, for 
the first time at the nth transition is denoted by 


A) := PIX, =5,Xn-1 #5,---,- X1#5|Xo=i] forn>1andi,j 20 
(3.24) 


Remarks. i) When z = 7, oe” is the probability of first return to the initial 
state 7 after exactly n transitions. 


ii) We have the following relation between po and pi”: 
. k 
k = 
A) = 3 Aoly® (625) 
k=1 


When the p's are known, we can use the preceding formula recursively to 


obtain the rae ’s, for & = 1,2,...,n. 


Example 3.2.2. In some cases, it is easy to calculate directly the probabilities 
of first return in n transitions. For example, let 
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0 1 
_ 0}1/3 2/3 
De | 1 0 | 
be the matrix of transition probabilities in one step of a Markov chain whose 


possible values are 0 and 1. We find that 


os) = P[Xy =1,Xn-1 =0,...,X1 =0| Xo = 0] = (1/3)"-1(2/3) = — 


for n = 1,2,... . Similarly, 


OQ =1 and po) =0 for n= 253, 3.5 


Next, we have 
pai = 1/3, Don = (2/3) «1 =2/3,.. “and pf} = =0 forn=3,4,... 


Finally, we calculate p;, (1) = 0, ep?) = 2/3, and 


forn = 3,4... 


_ 2 
py = 1x (1/3)"-7(2/3) = oo 


Note that, in this example, we have 77", pn i = 14 pan ) = 


Yio) =So 2-24 = 
0,1 — = 1” 
n=1 a0" 31-3 
and ze eo 
() 942 ce eae eee 
ei 3+ ge a O14 


These results hold true because, here, whatever the initial state, the process 


is certain to eventually visit the other state. 


Example 3.2.3. In Example 3.2.1, we calculated P(?). We also find that the 
matrix P() is given by 

0 1 2 

0 { 23/54 31/108 31/108 

P®) = 1] 5/12 7/24 7/24 

2| 4/9 5/18 5/18 


From this, we deduce from Eq. (3.25) that, for instance, 


1 1 1) (-1 i (1) 
B= POL = PIPL = Pox 1 => ol =5 
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and 
5 ) 1) (2-1 2) (2-2 
Fg = Pt = PoP + PU APLa 
Ll 2) (2) _ 
= 4X5 thx! => poi= 5 
and 
31 1)_(3-1 2) (3-2 3~3) 
BL 0) = ht + ple? + Ae’ 
eet eth a, Sy 4 
=aXqtgxgtrix! =? 0,1 = 57 
and so on. 


As in the preceding example, we can, at least for relatively small values of 
n, try to calculate directly pee Indeed, we have 
54 2) pf) x pit 
l=, 3° py 1 = py) X Po, 
and ‘hee 2 4 

(3) __ 1 (1 = = 
Po = Pop , X PEA X Phd + Bhd X Pao X Pha \=o7 +9 7 a7 

However, in general, the technique that consists in summing up the proba- 
bilities of all the paths leading from 7 to 7 in exactly 7 transitions rapidly 
becomes of little use. 


Until now, we have considered only conditional probabilities that the pro- 
cess {X,,n =0,1,...} finds itself in a state j at a time instant n, given that 
Xo =i. To be able to calculate the marginal probability P[X,, = j], we need 
to know the probability that Xo = i, for every possible state 7. 


Definition 3.2.3. The initial distribution of a Markov chain is the set 
{a;,i =0,1,...}, where a; is defined by 

= P[Xo = ti] (3.26) 
Remarks. i) In many applications, the initial position is deterministic. For 
example, we often suppose that Xo = 0. In this case, the initial distribution 


of the Markov chain becomes ag = 1 and a; = 0, for i = 1,2,... . In general, 
we have 0° )ai = 1. 


ii) The marginal probability ae i= P[X, = 3], for n = 1,2,..., is obtained 
by conditioning on the initial state: 


ay” = P[Xn = 3] = oPix n=Jj|Xo=@P[X = Yr a; (3.27) 


When ax = 1 for some k, we simply have ge = pe). 
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Example 3.2.4. Suppose that a; = 1/3, for i = 0,1, and 2, in Example 3.2.1. 
Then, we may write that 


and 


It follows that 


23 31 3131 
E[X2o] = 0x +1 x pe +2% Te = ag 
eng 31 31 155 
E{X2] = To Se? ee 
Pel= On x igg > * aas'= ig 
so that 3 
155/31 899 
Vike Say ee 
[Xa] = 39g (3) 1296 


Particular cases 


1) A (classic) random walk is a Markov chain whose state space is the set 
{...,—2,—-1,0,1,2,...} of all integers, and for which 


Pit = P= 1- Pii-1 for 7 = 0,+1,+2,... (3.28) 
for some p € (0, 1). Its one-step transition matrix is therefore (with g := 1—p): 


#-i1 @ £41442 


Pp 
4 (3.29) 


Thus, in the case of a classic random walk, the process can make a tran- 
sition from a given state 7 only to one or the other of its two immediate 
neighbors: 7 ~ 1 or i + 1. Moreover, the length of the displacement is always 
the same, i.e., one unit. If the state space is finite, say the set {0,1,...,N}, 
then these properties can be checked for each interior state: 1,2,...,N —1. 
When the process reaches state 0 or state N, there are many possibilities. For 
example, if 


Poo = PN,N =1 (3.30) 
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we say that the states 0 and N are absorbing. If po. = 1, the state 0 is said 
to be reflecting. 


Remarks. i) If p12 = p, pi. = ¢q, and pi = (1 —c)q, where 0 < c <1, the 
boundary at 0, when poo = 1, is said to be elastic. 


ii) We can also say that the process has a reflecting boundary at the point 
1/2 if the state space is {1,2,...}, pi =, and p12 =p. 


2) We can easily construct a Markov chain from a set Yo,Y1,... of i.id. 
random variables, whose possible values are integers, by proceeding as follows: 
let 


ia Vey Torta Un (3.31) 
k=0 


Then, {X,,,2 =0,1,...} is a Markov chain and, if 


Oe =F SFll Vk (3.32) 
we have 
n+1 n 
pigt=P[Xng1 =F (|Xn=T=P|OON=F NI (3.33) 
k=0 k=0 
nm 
=P Yrer =Jj-it S_ Yi PlYati = Jj — it] = 05-3 
k=0 


We say that the chain, in addition to being time-homogeneous, is also homo- 
geneous with respect to the state variable, because the probability p;,; does 
not depend explicitly on ¢ and 7, but only on the difference 7 — 7. 


3.2.2 Properties 


Generally, our first task in the study of a particular Markov chain is to de- 
termine whether, from an arbitrary state i, it is possible that the process 
eventually reaches any state j or, rather, some states are inaccessible from 
state 7. 


Definition 3.2.4. We say that the state 7 is accessible from i if there exists 
ann >0 such that pi” > 0. We write i j. 


Remark. Since we include n = 0 in the definition and since p = 1> 0, any 
state is accessible from itself. 


Definition 3.2.5. If i is accessible from 7, and j is accessible from t, we say 
that the states t and 7 communicate, or that they are communicating. We 
write 1 j. 
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Remark. Any state communicates with itself. Moreover, the notion of com- 
munication is commutative: 1 j => j © i. Finally, since (by the Chapman- 
Kolmogorov equations) 


poe > pp) for all m and n > 0 (3.34) 


we can assert that this notion is also transitive: if i @ j and j « k, then 
i k. Actually, it is sufficient to notice that if it is possible to move from 
state i to state 7 in m steps, and from j to k in n steps, then it is possible 
(by the Markov property) to go from i to k in m +n steps. 


Definition 3.2.6. [fi «+ j, we say that the states i and j are in the same 
class. 


Remark. The transitivity property (see above) implies that two arbitrary 
classes are either identical or disjoint. Indeed, suppose that the state space is 
the set {0,1,2} and that the chain has two classes: {0,1} and {1,2}. Since 
0+ 1 and 1 © 2, we have that 0 — 2, which contradicts the fact that the 
states 0 and 2 are not in the same class. Consequently, we can decompose the 
state space into disjoint classes. 


Definition 3.2.7. Let C be a subset of the state space of a Markov chain. We 
say that C is a closed set if, from any i € C, the process always remains in 
C: 


P[Xn41 €C |X, =iECl=1 forallieC (3.35) 


Definition 3.2.8. A Markov chain is said to be irreducible if all the states 
communicate, that is, if the state space contains no closed set apart from the 
set of all states. 


The following result is easy to show. 


Proposition 3.2.2. [fi — j or j — i for all pairs of states i and j of the 
Markov chain {X,,n =0,1,...}, then the chain is irreducible. 


Remark. We can also give the following irreducibility criterion: if there exists 
a path, whose probability is strictly positive, which starts from any state 7 
and returns to 7 after having visited at least. once all other states of the chain, 
then the chain is irreducible. We can say that there exists a cycle with strictly 
positive probability. 


Example 3.2.5. The Markov chain whose one-step transition matrix P is given 
in Example 3.2.1 is irreducible, because we may write that 
1 


x x A ee equate aug 
Po,1 X P1,2 P20= 3 3 a, 
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Note that here the cycle 0 — 1 — 2 — 0 is the shortest possible. However, we 
also have, for example, 


Po,2 X P2,0 X Po, X P1,2 X P2,0 > 0 
Remark. If a Markov chain (with at least two states) has an absorbing state, 
then it cannot be irreducible, since any absorbing state constitutes a closed 


set by itself. 


Definition 3.2.9. The state i is said to be recurrent if 


fii = P LJ {Xn = i} 


n=1 


Xn | =1 (3.36) 


If fis <1, we say that i ts a transient state. 


Remarks. i) The quantity f;; is the probability of an eventual return of the 
process to the initial state i. It is a particular case of 


fig = P 


Weenie | (3.37) 


which denotes the probability that, starting from state i, the process will 
eventually visit state 7. We may write that 


CO 
fig = 3 Dey (3.38) 
n=1 
ii) It can be shown that the state space Sx, of a Markov chain can be decom- 
posed in a unique way as follows: 


Sx, =DUC\UC2ZUC3... (3.39) 


where D is the set of transient states of the chain and the C,’s, k = 1,2,..., 
are closed and irreducible sets (that is, the states of each of these sets com- 
municate) of recurrent states. 


Proposition 3.2.3. Let N; be the number of times that state i will be visited, 
given that Xo =i. The state i is recurrent if and only if E[N;] = 00. 


Proof. We have 
PIN} ala fie (fa). for wa, yee: (3.40) 


That is, Nj ~ Geom(p := 1 — fj,;). Since E[N,| = 1/p, we indeed have 
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E[Ni] =o = fix=1 OO (3.41) 


~ 1 fis 


Remark. Since the set T = {0,1,...} is countably infinite, that is, time con- 
tinues forever, if the probability of revisiting the initial state is equal to 1, 
then this state will be visited an infinite number of times (because the process 
starts anew every time the initial state is visited), so that P[N; = oo] = 1, 
which directly implies that E[N;] = oo. Conversely, if 7 is transient, then we 
have ; 


P[N; = 00] = lim P{[N; >n] = lim ff>1 =0 (3.42) 
n-0o noo 


That is, a transient state will be visited only a finite number of times. Conse- 
quently, if the state space of a Markov chain is finite, then at least one state 
must be recurrent (otherwise no states would be visited after a finite time). 


Proposition 3.2.4. The state i is recurrent if and only if ae ph =O. 


Proof. Let I;x,, =i} be the indicator variable of the event {Xn = i}. That is, 


_ Jlif XxX, =i 
We only have to use the preceding proposition and notice that, given that 
Xo = i, the random variable I;x,,=i} has a Bernoulli distribution with pa- 
rameter p := P[|X, =i| Xo = i], so that 


CO CO 
E(Nj] = E bp Itx,,=9|Xo = | => Eltx,=i | Xo =i] 
n=0 


n=0 


Me 


i,t 


Co 
P[Xp 4) Mo =i] =) om a (3.44) 
n=0 


Remark. It can also be shown that if 7 is transient, then 


oo 
Ss; pe? <oo for all states k 
n=1 


The next result is actually a corollary of the preceding proposition. 


Proposition 3.2.5. Recurrence is a class property: if states i and j commu- 
nicate, then they are either both recurrent or both transient. 
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Proof. It is sufficient to show that if i is recurrent, then j too is recurrent 
(the other result being simply the contrapositive of this assertion). 

By definition of communicating states, there exist integers k and | such 
that pi) > 0 and pW > 0. Now, we have 


Pw 

lintk (l k 

pr Sp ree ow (3.45) 
Then 
(lintk 1) (k 
Seals > only oo? = = (3.46) 
> aera cle 
= 00 


Thus, 7 is recurrent. 0 


Corollary 3.2.1. All states of a finite and irreducible Markov chain are re- 
current. 


Proof. This follows indeed from the preceding proposition and from the fact 
that a finite Markov chain must have at least one recurrent state (see p. 88). 
Q 


Notation. Let i be a recurrent state. We denote by p; the average number 
of transitions needed by the process, starting from state i, to return to 7 for 
the first time. That is, 


“> np? (3.47) 


Definition 3.2.10. Let i be a recurrent state. We say that i is 


positive recurrent if pi; < oo 
null recurrent if 4; = oo 


Remarks. i) It can be shown (see Prop. 3.2.4) that ¢ is a null recurrent state 


if and only if pene =0 Pi, 2 = oo, but limpoo B= = 0. We then also have 


Himy_+o0 Dee = 0, for all states k. 


ii) It can be shown as well that two recurrent states that are in the same 
class are both of the same type: either both positive recurrent or both null 
recurrent. Thus, the type of recurrence is also a class property. 


iii) Finally, it is easy to accept the fact that any recurrent state of a finite 
Markov chain is positive recurrent. 
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Example 3.2.6. Because the Markov chain considered in Example 3.2.1 is finite 
and irreducible, we can at once assert that the three states—0, 1, and 2—are 
positive recurrent. However, the chain whose one-step transition matrix is 
given by 
0 1 2 
0] 1/3 1/3 1/3 
P,;=1] 0 1/21/2 
2; 0 1 #0 


is not irreducible, because state 0 is not accessible from either 1 or 2. Since 
there is a probability of 2/3 that the process will leave state 0 on its very first 
transition and will never return to that state, we have 


foo <1-—2/3=1/3 <1 (actually, foo = 1/3) 


which implies that state 0 is transient. Next, 


1 
P12 X pai = 3 >0 


so that states 1 and 2 are in the same class. Because the chain must have at 
least one recurrent state, we conclude that 1 and 2 are recurrent states. We 
may write that Sx, = DUC, where D = {0} and C, = {1, 2}. 
When 
0 1 2 
0/ 1/3 1/3 1/3 
Pg=1} 0 1/21/2 
2} 0 0 1 


we find that each state constitutes a class. The classes {0} and {1} are tran- 


sient, whereas {2} is recurrent. These results are easy to check by using Propo- 
sition 3.2.4. We have 


i 
n=0 n=0 3 
co oo 
1 1 
oer = = aaa =2<00 
n=0 n=0 2 
and 

0° n) co 

do Pha = D1 =00 

n= n=0 


As we already mentioned, any absorbing state (like state 2 here) trivially 
constitutes a recurrent class. Moreover, the state space being finite, state 2 
is positive recurrent. Finally, we have that Sx, = DUCi, where D = {0,1} 
and Cy = {2}. 
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Definition 3.2.11. A state i is said to be periodic with period d if po =0 
for any n that is not divisible by d, where d is the largest integer with this 
property. If d = 1, the state is called aperiodic. 


Remarks. i) If pe? = 0 for all n > 0, we consider 7 as an aperiodic state. That 
is, the chain may start from state 7, but it leaves 7 on its first transition and 
never returns to 2. For example, state 0 in the transition matrix 


O: a8 
of 0 1/21/2 
P=1/01/21/2 
2/0 0 1 


is such that ps = 0 for alln > 0. 

ii) As in the case of the other characteristics of the states of a Markov chain, 
it can be shown that periodicity is a class property: if 7 and 7 communicate 
and if 4 is periodic with period d, then j too is periodic with period d. 


iii) If pi) > 0, then 4 is evidently aperiodic. Consequently, for an irreducible 
Markov chain, if there is at least one positive element on the diagonal (from 
po,o) of the transition matrix P, then the chain is aperiodic, by the preceding 
remark. 


iv) If pe) > 0 and p? > 0, then 7 is aperiodic, because for any n € {2,3,... } 
there exist integers ky and kp € {0,1,...} such that n = 2k, + 3kp. 


v) If d = 4 for an arbitrary state i, then d = 2 too satisfies the definition of 
periodicity. Indeed, we then have 


ries = 0° -for #= 0,12. 


so that we can assert that py? = 0 for any n that is not divisible by 2. Thus, 
we must really take the largest integer with the property in question. 


Example 3.2.7. lf 
01 


of01 
Peay al 


then the Markov chain is periodic with period d = 2, because it is irreducible 


and 
oraaes =0 and pen aD efor WSO 1 usd 
Note that d is not equal to 4, in particular because 2 is not divisible by 4 and 


py) =1>0. 
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Example 3.2.8. Let 
0 1 2 
O| 0 1/21/2 
P=1]/1/2 0 1/2 
211/21/2 0 
The Markov chain is aperiodic. Indeed, it is irreducible (since po 1 Xp1,2*P2,0 = 
1/8 > 0) and 
py) (=1/2) >0 and pi} (=1/4) >0 


Example 3.2.9. Let 


01 2 3 4 
of01/21/2 0 0 
1/0 0 0 1/32/3 

P=2|0 0 0 2/31/3 
3/10 0 0 0 
4410 0 0 0 


be the one-step transition matrix of a Markov chain whose state space is the 
set {0,1, 2,3, 4}. We have 


Po, X P,3 X P3,0 X Po,2 X P2,4 X pao > O 
Therefore, the chain is irreducible. Moreover, we find that 


(n) _ flifn=0,3,6,9,... 
9,0 ~ | 0 otherwise 


Thus, we may conclude that the chain is periodic with period d = 3. 


Example 3.2.10. In the case of a classic random walk, defined on all the inte- 
gers, all the states communicate, which follows directly from the fact that the 
process cannot jump over a neighboring state and that it is unconstrained, 
that is, there are no boundaries (absorbing or else). The chain being irre- 
ducible, the states are either all recurrent or all transient. We consider state 
0. Since the process moves exactly one unit to the right or to the left at each 
transition, it is clear that if it starts from 0, then it cannot be at 0 after an 
uneven number of transitions. That is, 


ie =0 forn=0,1,... 


Next, for the process to be back to the initial state 0 after 2n transitions, there 
must have been exactly n transitions to the right (and thus n to the left). Since 
the transitions are independent and the probability that the process moves to 
the right is always equal to p, we may write that, for n = 1,2,..., 


phe) = P[B(2n,p) = n] = (“ena —p)= Corn — p)” 
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To determine whether state 0 (and therefore the chain) is recurrent, we 


consider the sum 
SA (n) _ Y_(2n) 
do P00 = DP o0 
n=l n=1 


However, we do not need to know the exact value of this sum, but rather 
we need only know whether it converges or not. Consequently, we can use 
Stirling’s? formula: 


nin n tte Von (3.48) 


(that is, the ratio of both terms tends to 1 when n tends to infinity) to 
determine the behavior of the infinite sum. We find that 


(an) [4p(1 — p)}” 
0,0 TN 


which implies that 
CO [oe] 
(n) [4p(1 — p)|” 
Sipg <0o = SO SY <0 
n=1 n=l mm 


Now, if p = 1/2, the stochastic process is called a symmetric random walk 
and the sum becomes 


n=1 
because = = 
1 1 1 
> -= 
> T° St as n a 
n=l n=l 
When p # 1/2, we have 
coe) CO 
4p(1 — p)i” 1—p)\” 
So ao =a SS MODI 
pe Jin a Ji 


Thus, the classic random walk is recurrent if p = 1/2 and is transient if 
p# 1/2. 

Remark. It can be shown that the two-dimensional symmetric random walk 
too is recurrent. However, those of dimension k > 3 are transient. This follows 
from the fact that the probability of returning to the origin in 2n transitions 
is bounded by c/n*/?, where c is a constant, and 


oo 
c 

) <= <o ifk>3 (3.49) 
k —_ 

wai e 


2 James Stirling, 1692-1770, who was born and died in Scotland, was a mathemati- 
cian who worked especially in the field of differential calculus, particularly on the 
gamma, function. 
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3.2.3 Limiting probabilities 


In this section, we will present a theorem that enables us to calculate, if it 
exists, the limit limn_, ph” by solving a system of linear equations, rather 
than by trying to obtain directly lim,_.., P™, which is generally difficult. 
The theorem in question is valid when the chain is irreducible and ergodic, as 
defined below. 


Definition 3.2.12. Positive recurrent and aperiodic states are called ergodic 
states. 


Theorem 3.2.1. In the case of an irreducible and ergodic Markov chain, the 
limit 
— fj (n) 
a= jim Pi (3.50) 


exists and is independent oft. Moreover, we have 
1 
rec for all j € {0,1,...} (3.51) 
J 


where 4; is defined in Eq. (3.47). To obtain the m;’s, we can solve the system 


n= aP (3.52) 
foe] 
Soa =1 (3.53) 
j=0 


where m := (70,74,.-.). It can be shown that the preceding system possesses 
a unique positive solution. 


Remarks. i) For a given state j, Eq. (3.52) becomes 


oo 
ui= Som Pi,j (3.54) 
i=0 
Note that, if the state space is finite and comprises k states, then there are k 
equations like the following one: 


k-1 
p=. Dg (3.55) 
= 


With the condition pa, m; = 1, there are thus k + 1 equations and k un- 
knowns. In practice, we drop one of the k equations given by (3.55) and make 
use of the condition Re: mj = 1 to obtain a unique solution. Theoretically, 
we should make sure that the 7;’s obtained satisfy the equation that was 
dropped. 
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ii) A solution {7;,j = 0,1,...} of the system (3.52), (3.53) is called a station- 
ary distribution of the Markov chain. This terminology stems from the fact 
that if we set 


then, proceeding by mathematical induction, we may write that 


[o@) 
= or T= Tj (3.57) 
i=0 
where the last equality follows from Eq. (3.54). Thus, we find that 
PIX, =jl=a, Vn j € {0,1,2,...} (3.58) 


iii) If the chain is not irreducible, then there can be many stationary distri- 
butions. For example, let 


Because states 0 and 1 are absorbing, they are positive recurrent and aperi- 
odic. However, the chain has two classes: {0} and {1}. The system that must 
be solved to obtain the 7;’s is 


To = 70 (3.59) 
Ty) = (3.60) 
mo+m™ =1 (3.61) 


We see that 7 = c, 7; = 1—c is a valid solution for any c € (0, 1]. 


iv) The theorem asserts that the 7;’s exist and can be obtained by solving 
the system (3.52), (3.53). If we assume that the 7;’s exist, then it is easy to 
show that they must satisfy Eq. (3.52). Indeed, since limpsoo P[Xn4i = j] = 
liMpsoo P[X, = j], we have 


P[Xnti = jj] =: PiXna =j|Xn = P(X, = i] 


7=0 
=> lim PiXnei =a) = sis Does =j|Xn=4P[X, =i] 


oO oO 
=> WSs S°Pi,j m= So mi i,j (3.62) 
1=0 i=0 
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where we assumed that we can interchange the limit and the summation. 


v) In addition to being the limiting probabilities, the 7;’s also represent the 
proportion of time that the process spends in state 7, over a long period of 
time. Actually, if the chain is positive recurrent but periodic, then this is the 
only interpretation of the 7;’s. Note that, on average, the process spends one 
time unit in state j for yj time units, from which we deduce that 7; = 1/p;, 
as stated in the theorem. 


vi) When the Markov chain can be decomposed into subchains, we can ap- 
ply the theorem to each of these subchains. For example, suppose that the 
transition matrix of the chain is 


01 2 
of/i1 0 0 
P = 1]01/21/2 
2|01/21/2 
In this case, the limit limp_,o¢ oy exists but is not independent of i. We easily 
find that 
, (n)_ _flifz=0 
a= ‘ ifi £0 ee) 


and (by symmetry) 


: Vo pe 
gu Pay Pe 9 tO oie) 


(n Oona 
However, we cannot write that 7) = 1 and m = m2 = 1/2, since the 7;’s do 
not exist in this example. At any rate, we see that the sum of the 7;’s would 
then be equal to 2, and not to 1, as required. 


Example 3.2.11. The Markov chain {X,,n = 0,1,...} considered in Exam- 
ple 3.2.1 is irreducible and positive recurrent (see Example 3.2.6). Since the 
probability po,o (or pi,1) is strictly positive, the chain is aperiodic, and thus 
ergodic, so that Theorem 3.2.1 applies. We must solve the system 


1/3 1/3 1/3 
(70,71, %2) = (m0, 71,72) | O 1/2 1/2 
1 0 0 
that is, 
To @) eae 
3 
Ty 2 amy bam 
3 2 
(3) 1 
™ = 37+ 5m 
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subject to the condition 79 + 7 + 72 ©) 1. Often, we try to express every 
limiting probability in terms of 79, and then we make use of the condition (*) 
to evaluate 7. Here, Eq. (1) implies that 72 = 270, and Eq. (2) enables us 


3 
to assert that 7, = 279 too [so that 7, = #2, which actually follows directly 


3 
from Eqs. (2) and (3)]. Substituting into Eq. (*), we obtain: 
To + + =A J > 7 oe 
oF 370 T 3% = > oO 7 1=72= 7 


We can check that these values of the 7;’s are also a solution of Eq. (3), which 


was not used. 
Note finally that the 2;’s correspond to the limits of the pe 
in Example 3.2.1 by finding the approximate value of the matrix P() when 


n is large. 


s computed 


When the state space Sx, is finite, before trying to solve the system (3.52), 
(3.53), it is recommended to check whether the chain is doubly stochastic (see 
p. 79), as can be seen in the following proposition. 


Proposition 3.2.6. In the case of an irreducible and aperiodic Markov chain 
whose state space is the finite set {0,1,... ,k}, if the chain is doubly stochastic, 
then the limiting probabilities exist and are given by 


1 : 

i ee Se forj =0,1,...,k (3.65) 
Proof. Since the number of states is finite, the chain is positive recurrent. 
Moreover, as the chain is aperiodic (by assumption), it follows that it is er- 
godic. Thus, we can assert that the 7;’s exist and are the unique positive 
solution of 


k 
m= > mpig VI € {0,1,... ,k} (3.66) 
i=0 
k 
j=0 
Now, if we set 7; = 1/(k +1) > 0, we obtain 
k 
1 
—— =] 3.68 
Sess eee 
j=0 
and 
k k 
1 1 1 1 
i = x 1 = — 3.69 
Deas bri LPs k+l k+1 ne 


(because the chain is doubly stochastic). 0 
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Example 3.2.12. The matrix 


0 1 2 
0 1/3 1/3 1/3 
P=1] 0 1/21/2 
2} 2/3 1/6 1/6 
is irreducible and aperiodic (because pp,o > 0, in particular). Since it is doubly 
stochastic, we can conclude that 7; = 1/3 for each of the three possible states. 
We can, of course, check these results by applying Theorem 3.2.1. 


Example 3.2.13. The one-step transition matrix 


Or 2 
of1 0 0 
PS tT01/2 172 
2}01/21/2 


is doubly stochastic. However, the chain is not irreducible. Consequently, we 


cannot write that 7; = 1/3. Actually, we calculated the limits limn—oo py in 
remark vi) on p. 96. 


When the state space {0,1,...} is infinite, the calculation of the limiting 
probabilities is generally difficult. However, when the transition matrix is such 
that poi > 0, Pj,j-1 X Pjj+1 > OV 7 > 0, and 


Pig =O if|j-—i[>1 (3.70) 
we can give a general formula for the 7;’s. Indeed, we have 
To = ToPo,o + 71P1,0 (3.71) 
and 
Tj = 1j-1Pj—1,5 + ™5D5,5 + 541Pj 41,5 (3.72) 
for 7 = 1,2,.... We find that 


APA SPI OS Pilg 


= for 7 = 1,2,... (3.73 
Pi,o X P21 X +°* X Pj,j-1 : ) 


Tj 


Then a necessary and sufficient condition for the existence of the 7,’s is that 


lee) 
x > a4 
PO,1 Pi,2 Pk-1,k < (3.74) 


fen P10 xX po X ++: X Dkyk-1 


If the sum above diverges, then we cannot have that Pe, Tj =1. 
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Note that this type of Markov chain includes the random walks on the 
set {0,1,...}, for which poi = p = 1— poo and pii41 = Pp = 1— piy-1, for 
i= 1,2,.... We have 

> Po,1 X Pia X ++ X p = 
0,1 1,2 oe k—-1,k = > (3.75) 


ise 
427 Plo X p2,. X +++ X Pkyk—1 (l= p) 


k=1 


and this sum converges if and only if p < 1/2. Thus, we can write (with 
q:=1-—p) that 


oo 


mo )(p/gt=1 = m=1-% (3.76) 
k=0 q 
and then 
j j 
q q q 
for 7 =1,2,.... 


Actually, we could have used Theorem 3.2.1 to obtain these results. Indeed, 
consider the transition matrix 


Oo 12 B catia ale k-2k-1k 
3) qp 
1 |q0 p 
2 q 0 p 
P=: Nera oe (3.78) 
kt q Op 
k q P 


As the Markov chain is irreducible and ergodic, we can try to solve the system 


T = 4G TOT MH 
7 =pj-1t+¢@ 741 forj=1,... »k-1 


Tk =P Te-1 + P Tr 


We find that 


2 
i= (2) To for j = i eee . kK (3.79) 
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Then, the condition ar 1; = 1 enables us to write 


a) 


Taking the limit as k tends to infinity, we obtain (in the case when p< q & 


p< 1/2) 
™ = ( = 2) and 1 (2) (1 = 4 (3.81) 


for 7 = 1,2,..., which effectively corresponds to the formulas (3.76) and 
(3.77). 


3.2.4 Absorption problems 


We already mentioned (see p. 87) that the state space of a Markov chain can 
be decomposed into the set D of transient states of the chain and the union 
of closed and irreducible sets C, of recurrent states. We are interested in the 
problem of determining the probability that, starting from an element of D, 
the process will remain indefinitely in D or instead will enter one of the sets 
Cx, from where it cannot escape. 


Notation. Let i € D and let C be a recurrent class. We set 


n(C) = PIX, €C|Xo=i => ph) forn=1,2,... (3.82) 
gEc 


and 


ri(C) = lim rM(C) = r|U es 


=| (3.83) 


=1 
We have the following result. 


Theorem 3.2.2. The probability r;(C) is the smallest nonnegative solution 
of the system 


= > mgts(C)+> p53 for allie D (3.84) 
jED geCc 
Moreover, if D is a finite set, then the solution is unique. 
Remarks. i) The system (3.84) is a system of nonhomogeneous linear equa- 


tions. When D is finite, we can try to solve this system by using results from 
linear algebra. 


3.2 Discrete-time Markov chains 101 


ii) Actually, it is not necessary that the set C’ be a class. It is sufficient that 
C be a closed set of recurrent states. 


iii) We can imagine that all the states of the recurrent class C constitute a 
single absorbing state, since, once the process has entered this class, it cannot 
leave it again. 


A classic example of an absorption problem is known as the gambler’s 
ruin problem and is described as follows: a player, at each play of a game, 
Wins one unit (for example, one dollar) with probability p and loses one unit 
with probability g := 1—p. Assume that he initially possesses 7 units and that 
he plays independent repetitions of the game until his fortune reaches k& units 
or he goes broke. 

Let X,, be the fortune of the player at time n (that is, after n plays). Then 
{Xn,n =0,1,...} is a Markov chain whose state space is the set {0,1,... , k}. 
As states 0 and k are absorbing, we have that poo = px,~ = 1. For all the other 
states i= 1,...,k—1, we may write that 


Pitt =Pp=1—pii~r (3.85) 


This is a random walk on the set {0,1,... ,k}, with absorbing boundaries at 
0 and k. The chain thus has three classes: {0}, {k}, and {1,2,...,4—1}. The 
first two are recurrent, because 0 and k are absorbing, whereas the third one 
is transient. Indeed, we have, in particular, that 


P[X; =0| Xo =1])=q>0 => fit <1 (3.86) 


from which we can conclude that the player’s fortune will reach 0 or k units 
after a finite number of repetitions. 

Let r;({0}), for i = 0,1,...,k, be the probability that the player will be 
ruined, given that his initial fortune is equal to 7 units. That is, we write that 
C is the class {0} in Eq. (3.83). We will first consider the case when p = 1/2. 
Note that, in this case, we have 


E[X, | Xo= i] = (i) x +41) x5 <3 fori=1,...,k—1 (3.87) 
We also have 
E[X,| Xo =0])=0 and E[X,| Xo =k] =k (3.88) 
That is, in general, 
E{Xn4i| Xn =i] =i for anyi (3.89) 


This type of Markov chain is a martingale and is very important for the 
applications, notably in financial mathematics. 
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Definition 3.2.13. A Markov chain for which 


E[Xnvi|Xn=i)= D> jpig =i for allie Sx, (3.90) 
JESxy 


is called a martingale. 


Remark. We can rewrite Eq. (3.90) as follows: 
E[Xn+1 | Xn] = Xp (3.91) 


Now, proceeding by induction, we may write that 
k 
= E[Xn|Xo=i}= > jo (3.92) 
j=0 


Indeed, if we make the induction assumption that E[Xn-1 | Xo = 1] = 7, then 
we have 


k ok 
. n n—-1 
E[Xn | Xo =i) = Sip) = S> So ivi hy? 
j=0 j=0 l=0 
k 
1 
Savi = reer n-1|Xo=]] 
i=0 a0 1=0 
k 
=) > pil = E[X1 | Xo =i] = 4 (3.93) 
1=0 
Now, we have 
im, py ") 0 for all transient states j (3.94) 


(otherwise the sum 7°, ps a would diverge, contradicting the remark on 
p. 88). It follows, taking the limit as n tends to infinity in Eq. (3.92), that 


i= lim. onl + Sip +k $ =k Jim DPA (n) (3.95) 
j=l 
That is, 
eee 3.96 
Sue Pik k ( ) 


from which we deduce that 


si gt cay ext (oyste 97 
Jim pio =1 5 = ri({O}) =1 i (3.97) 
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Thus, the more ambitious the player is, the greater is the risk that he will be 
ruined. 


In the general case where p € (0,1), the probability r;({0}), which will be 
denoted by r; to simplify the formulas, is such that ro = 1, rz, = 0 and [see 
Eq. (3.84)| 


r= pre+q (3.98) 
re=qri-itprigi: fori =2,...,k—2 (3.99) 

and 
Tk-1 = QTk-2 (3.100) 


Rewriting Eq. (3.99) as follows: 
(p+q@)r =4ri-1t Priva (3.101) 
we obtain 


Ti41 TE = (Ty — Ti-1) for 1 = 2,... »k-2 (3.102) 


SB 1a 


which implies that 


Tigi —Ti = en (rg —11) = (2) (r1 -1) (3.103) 


where the last equality follows from Eq. (3.98). 
Next, since r;, = 0, Eq. (3.100) can be rewritten as follows: 


Tk-1 = QT k-2 + PP (3.104) 


We can then state that Eq. (3.103) is valid for i = 0,...,4 —1. Adding the 
equations for each of these values of 7, from 0 to 7 — 1, we find that 


ee Ce SCT (3.105) 


v 


ll 
° 


Since rg = 1, we obtain 


1- (apy, 
pe Lape eee (3.106) 


1+ (ri —1) ifp=q 


for j = 0,1,...,k. Finally, the fact that r, = 0 enables us to obtain an explicit 
expression for r; — 1 from the preceding formula, from which we find that 
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oe Sg 
r= Pee (3.107) 
1- ifp=4q 


Note that for p = q = 1/2, we retrieve the formula (3.97). 
We can calculate the limit of the probability r; in the case when k tends 
to infinity. It is easy to check that 


(q/p)? if p > 1/2 3 
a .108 
ae { 1 ifp<1/2 t9:108) 
Thus, if p > 1/2, there exists a strictly positive probability that the process 
will spend an infinite time in the set of transient states. 


Another example for which the Markov chain may spend an infinite time 
in the set D of transient states of the chain is the following. 


Example 3.2.14. Let poo = 1 and 
P30 = 9; (>0) =1—pjjy1 for j =1,2,... 


be the one-step transition probabilities of a Markov chain whose state space 
is the set {0,1,...}. This chain has two classes: {0} (recurrent) and {1,2,...} 
(transient). We calculate directly 


oOo 


ry({0}) = 1 — [J (1 - a5 44) 
i=0 
It can be shown that r;({O}) < 1 if and only if the sum 5772, aj4: converges. 
For example, if a; = (1/2)*, for all i > 1, then we have 


Saye = S(1/2# = (1/29? < 00 
i=0 i=0 


3.2.5 Branching processes 


In 19th-century England, some people got interested in the possibility that 
certain family names (particularly names of aristocratic families) would dis- 
appear, for lack of male descendants. Galton® formulated the problem math- 
ematically in 1873, and he and Watson‘ published a paper on this subject in 


3 Francis Galton, 1822-1911, was born and died in England. He was a cousin of 
Charles Darwin. After having studied mathematics, he became an explorer and 
anthropologist. 

4 The Reverend Henry William Watson, 1827-1903, was born and died in England. 
He was a mathematician who wrote many books on various subjects. He became 
a priest in 1858. 
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1874. Because Bienaymé® had worked on this type of problem previously, the 
corresponding stochastic processes are sometimes called (branching) processes 
of Bienaymé-Galion—Watson. More simply, the term branching processes is 
also used. 


Definition 3.2.14. Let {Z,,;,n =0,1,... 37 =1,2,...} be a set of itd. ran- 
dom variables whose possible values are nonnegative integers. That is, Sz, , C 
{0,1,...}. A branching process is a Markov chain {Xn,n = 0,1,...} de- 
fined by 


x, <) dy Fntj if Xn-1 > 0 (3.109) 


forn=1,2,.... 


Remarks. i) In the case of the application to the problem of the disappearance 
of family names, we can interpret the random variables X, and Zy~1,; as 
follows: Xo is the number of members of the initial generation, that is, the 
number of ancestors of the population. Often, we assume that Xo = 1, so that 
we are interested in a lineage. Zy_1,; denotes the number of descendants of 
the jth member of the (n — 1)st generation. 


ii) Let 
Pii= P(Zp-1,5 = 1] for all n and j (3.110) 


To avoid trivial cases, we assume that p; is strictly smaller than 1, for all 
7 = 0,1,.... We also assume that pp > 0; otherwise, the problem of the 
disappearance of family names would not exist. 

The state space Sx, of the Markov chain {X,,n = 0,1,...} is the set 
{0,1,...}. As state 0 is absorbing, we can decompose S'x,, into two sets: 


Sx, = Dv {0} (3.111) 


where D = {1,2,...} is the set of transient states. Indeed, since we assumed 
that po > 0, we may write that 


PIXn fiV ne {1,2,...} | Xo =i] > pio M pi >0 (3.112) 


Thus, fi <1, and all the states i = 1,2,... are effectively transient. Now, 
given that a transient state is visited only a finite number of times, we can 


5 Trénée-Jules Bienaymé, 1796-1878, was born and died in France. He studied at 
the Ecole Polytechnique de Paris. In 1848, he was named professor of probability 
at the Sorbonne. A friend of Chebyshev, he translated Chebyshev’s works from 
Russian into French. 
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assert that the process cannot remain indefinitely in the set {1,2,...,k}, for 
any finite k. Thus, we conclude that the population will disappear or that its 
size will tend to infinity. 

Suppose that Xo = 1. Let’s now calculate the average number j, of indi- 
viduals in the nth generation, for n = 1,2,... . Note that 44; = E[X)] is the 
average number of descendants of an individual, in general. We have 


CO 
Hn = E[Xn] = D> E[Xn | Xn-1 = j|P[Xn-1 = J] 
j=0 
oo 
=) jm P[Xn-1 = J] = mM E[Xn-1] (3.113) 
j=90 


Using this result recurrently, we obtain 


Un = WE[Xp—1] = wiE[Xn-2] =... = WPE[Xo] = wt (3.114) 


Remark. Let of := V[X,]. When p = 1, from Eq. (1.96), which enables us 
to express the variance of X, in terms of E[X, | Xn—.] and of V[X;, | Xn-il, 
we may write that 


V(X] = E[V[Xn | Xn-i]] + VIE[Xn | Xn-1)] 
ids FX ,-102] + V[Xn-1 x UJ 
= 02x14 V[Xp-1] = 207 + V[Xn-2] 


. = noz+V[Xo] = not (3.115) 


since V[ Xo] = 0 (Xo being a constant). When s 4 1, we find that 


R-1 
ViXe) = oft (=) (3.116) 


We wish to determine the probability of eventual extinction of the popu- 
lation, namely, 


q0,i c= lim PiXy =0 | Xo = i| (3.117) 
n—-oo 
By independence, we may write that 


9,1 = %,1 (3.118) 


Consequently, it suffices to calculate go,,, which will be denoted simply by qo. 
We have 


P[X, =0| Xo =1) =1-—P[X, >1|Xo=1=1- 5) P[Xn =k| Xo = 1) 
k=1 


Il 
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CO 
>1-SORP{X, =k | Xo = 1] =1-E[X, | X=] 
k=1 
=1-p? (3.119) 


[by Eq. (3.114)]. It follows that, if 4; € (0,1), then 
go> limi-pt=1 = qg=l (3.120) 
n->0O 


which is a rather obvious result, since if each individual has less than one 
descendant, on average, we indeed expect the population to disappear. 

When f41 > 1, Eq. (3.119) implies only that go > 0 (if wi = 1) or that 
go = —0o (if uw, > 1). However, the following theorem can be proved. 


Theorem 3.2.3. The probability qo of eventual extinction of the population 
as equal to 1 if fi <1, while qo < 1 if uy > 1. 


Remarks. i) We deduce from the theorem that a necessary condition for the 
probability go to be smaller than 1 is that p; must be greater than 0 for at 
least one j > 2. Indeed, if pp = p > 0 and p; = 1 — p, then we directly have 
fy =1-p<l. 

ii) When po = 0 and p; = 1, we have that p, = 1. According to the theorem, 
we should have go = 1. Yet, if p, = 1, it is obvious that the size of the 
population will always remain equal to Xo, and then gg = 0. However, the 
theorem applies only when po > 0. 


Let F' be the event defined by 


F= UJ {X, = 0} (3.121) 


n=1 


so that go = P[F' | Xo = 1]. To obtain the value of go, we can solve the 
following equation: 


oO CO 
go = > PIF | Xi = J)p; = Dap; (3.122) 
j=0 j=0 


The equation above possesses many solutions. It can be shown that when 
Lt > 1, go is the smallest positive solution of the equation. 


Remark. Note that go = 1 is always a solution of Eq. (3.122). 


Example 3.2.15. Suppose that pp = 1/3 and p; = 2/9, for j = 1,2,3. First, 
we calculate 


2 4 
My =0+5(1+2+3)=3>1 


Thus, we may assert that go < 1. Eq. (3.122) becomes 
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oe, Fk 
g=ztGotata) = wta-59+5 =0 


Since go = 1 is a solution of this equation, we find that 
. 3 
(g0 — 1) { go + 240-5) =0 


The three solutions of the equation are 1 and —1 + 1/5/2. Therefore, we 
conclude that gg = —1 + 1/5/2 ~ 0.5811. 
Remark. If pp = 1/2 and p; = 1/6, for j = 1,2,3, then we have 


1 
jy = OF 5(1 4243) =1 
Consequently, Theorem 3.2.3 implies that gg = 1. Eq. (3.122) is now 


1 1 
go = 5 + Gg +95 +90) <> (qo—1)?(go +3) =0 


so that the three solutions of the equation are 1 (double root) and —3, which 
confirms the fact that go = 1. 
Similarly, if pp = pi = 1/2, then we find that 


1 
go = 5(1 +4) = g=l 
in accordance with Theorem 3.2.3, since 4 = 1/2 <1. 


Assume again that Xo = 1. Let 
Tr = > Xe =14+> Xe (3.123) 
k=0 k=] 


That is, T,, designates the total number of descendants of the population’s 
ancestor, in addition to the ancestor himself. It can be shown that 


ae [jim Tn z i] = (3.124) 


Thus, if gg < 1, then Tyo := limyn_.o. T;, is a random variable called defective, 
which takes the value oo with probability 1 — qo. Moreover, if gg < 1, the 
mathematical expectation of T,, is evidently infinite. When go = 1, we have 
the following result. 


Proposition 3.2.7. The mathematical expectation of the random variable T,, 
is gwen by 


E[T.] = ~ if <1 (3.125) 
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Remark. We observe that when 4; = 1, we have that P[T,. < oo] = 1, but 
ElT 3] = 00. 


Example 3.2.16. If po = p € (0,1) and p, = 1 — p, then we know that qo = 1, 
because fy = 1 — p< 1. In this case, we may write that 


xX. = 1 with probability (1 — p)* 
k ~ | 0 with probability 1 — (1 —p)* 


Thus, we have 


which is indeed equal to 1/(1 — 11). 
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3.3.1 Exponential and gamma distributions 


In the case of discrete-time Markov chains, we said nothing about the time 
the processes spend in state 7 before making a transition to some state j. As 
in Example 2.1.1 on random walks, we may assume that this time is determin- 
istic and is equal to one unit. On the other hand, an essential characteristic of 
continuous-time Markov chains is that the time that the processes spend in a 
given state has an exponential distribution, so that this time is random. More- 
over, we will show that the sum of independent exponential random variables 
(having the same parameter) is a variable having a gamma distribution. We 
already mentioned the exponential and gamma distributions in Chapter 1. In 
the present section, we give the main properties of these two distributions. 


Exponential distribution 


Definition 3.3.1. (Reminder) [f the probability density function of the con- 
tinuous random variable X, whose set of possible values ts the interval {0, 00), 
is of the form 


AL Gf, 
fx(x) = 5 fe : : (3.126) 


we say that X has an exponential distribution with parameter \ > 0 and 
we write X ~ Exp(r). 


Remarks. i) Using the Heaviside function u(x) (see p. 11), we may write that 


fx(z) =Ae*" u(x) (Vx ER) (3.127) 
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Note that we have the following relation between the functions u(x) and 6(z) 
(see p. 63): 


u(x) = a 6(t) dt (3.128) 


ii) For some authors, a random variable X having an exponential distribution 
with parameter possesses the density function 


fx(z) = xe u(t) (3.129) 


The advantage of this choice is that we then have E[X] = A, while for us 
E|[X] = 1/4, as will be shown further on. 


The distribution function of X is given by 


0 ifz <0 
= iy 
Fx(z) = / rhe ** dt =1-—e7-** ifa >0 (3.180) 
0 
Note that we have the following very simple formula: 
P[X>a]=e"" forr>0 (3.131) 


Some people write F(x) for the probability P[X > a]. 

The main reason for which the exponential distribution is used so much, in 
particular in reliability and in the theory of queues, is the fact that it possesses 
the memoryless property, as we show below. 


Proposition 3.3.1. (Memoryless property) Suppose that X ~ Exp()). 
We have 


P[X >s+t|X >t)=P[X>s] Vs,t20 (3.132) 


Proof. By the formula (3.131), we may write that 


PIX >s+t,X >t] Pix >s+i] 
P[X > t] ~ PIX >¢ 
e7X(stt) 


= ane = e’=P[X>s] O (3.133) 


PIX >s+t|X>t= 


Remark. Actually, the exponential random variables are the only r.v.s having 
this property for all nonnegative s and t. The geometric distribution possesses 
the memoryless property, but only for integer (and positive) values of s and t. 


We can easily calculate the moment-generating function of the r.v. X ~ 
Exp(A): 
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Mx(t) = Ele*] = [etre ae = = ie <X (3.134) 
Then 
Mi(t) = aoa and M(t) = re (3.135) 
which implies that 
E[X] = Mx(0) = x: E[X?] = MX(0) = = and V[X]= 3 (3.136) 


Remark. Note that the mean and the standard deviation of X are equal. 
Consequently, if we seek a model for some data, the exponential distribution 
should only be considered if the mean and the standard deviation of the 
observations are approximately equal. Otherwise, we must transform the data, 
for example, by subtracting or by adding a constant to the raw data. 


Example 3.3.1. Suppose that the lifetime X of a car has an exponential dis- 
tribution with parameter A. What is the probability that a car having already 
reached its expected lifetime will function (in all) more than twice its expected 
lifetime? 


Solution. We seek 


2 1} 1 ae ee 
pix>3]x>5]=p[x> 5] =« ~ 0.3679 


Remark. The assumption that the lifetime of a car has an exponential distribu- 
tion is certainly not entirely realistic, since cars age. However, this assumption 
may be acceptable for a time period during which the (major) failure rate of 
cars is more or less constant, for example, during the first three years of use. 
Incidentally, most car manufacturers offer a three-year warranty. 


As the following proposition shows, the geometric distribution may be 
considered as the discrete version of the exponential distribution. 


Proposition 3.3.2. Let X ~ Exp(A) and Y := int(X)+1, where “int” des- 
ignates the integer part. We have 


PY SkhSe“yatse*) prea 12.2: (3.137) 
That is, Y ~ Geom(p := 1—e7*). 


Proof. First, since int(X) € {0,1,...}, we indeed have that Sy = {1,2,...}. 
We calculate 


PlY =k] = Plint(X) =k-1)=Plk-1<X <A] (3.138) 


112 3 Markov Chains 


k 
= de“ dr =~ e™* 
k-1 


k Evie be 
=e ABU (gray 


Remark. If we define the geometric distribution by 
PIY =k] =q*p for k=0,1,2,... (3.139) 


(as many authors do), then we simply have that Y := int(X) ~ Geom(p = 
1—e-). 
We can also consider the exponential distribution on the entire real line. 


Definition 3.3.2. Let X be a continuous random variable whose density func- 
tron ts given by 


fx(z) = Seo Mel forxeER (3.140) 


where X is a positive parameter. We say that X has a double exponential 
distribution or a Laplace® distribution. 


Remark. We find that the mean value of X is equal to zero and that V[X] = 
2/\?. The fact that ELX] = 0 follows from the symmetry of the function fx 
about the origin (and from the existence of this mathematical expectation). 


Proposition 3.3.3. If X ~ Exp(A), then for all xo > 0, we have 


E[X |X >29]=20+ E[X] and V[X | X > xo] = V[X] (3.141) 


Proof. The formula P[X > zo] = e~**° implies that 
fx(z |X > 29) =A|e**-*) for x > x9 (3.142) 


from which we have 


CO CO 
E[X | X > 29] = if x de 2-20) dg YE i (y + 29)Ae7*Y dy 
0 


Zo 


= E[X] + 29P[X € (0, c0)] Ss E[X] + Zo (3.143) 


Next, we calculate 


CO oO 
E[X? | X > zo] = | ax? Ne Ale—20) da PE *° | (y + 2)*Ae7~*Y dy 
0 


= BX?) + 229 E[X] + 22P[X € [0,00)] 
= E[X?] + 2x9 E[X] + 2? (3.144) 


5 See p. 19. 
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Then we obtain 


V[X | X > ao] = E[X? | X > ao] — (E[X | X > 29])? 
= E[X?] — (E[X])? =V[X] oO (3.145) 


Remark. The preceding proposition actually follows directly from the memo- 
ryless property of the exponential distribution. 


A result that will be used many times in this book is given in the following 
proposition. 


Proposition 3.3.4. Let X; ~ Exp(\;) and X_ ~ Exp(Az2) be two independent 
random variables. We have 


M1 
PIX, < Xo] = 3.146 
Pi < 2] =p (3.146) 
Proof. By conditioning on the possible values of X;, we obtain 
CO 
P[X > X}] = ‘ P[X2 > X, | Xy = x) fx, (2) dx (3.147) 
0 


ind oO ioe) 
— , P[X_ > t]\ye™™"* dz = i: et) eM? dx 
0 0 


ioe) At 
= Nye A1tA2)2 de = oO 3.148 
| 1€ L Yt ( ) 


Remarks. i) We have that P[X, < X2] = 1/2 if A; = Ae, which had to be the 
case, by symmetry and continuity. 


ii) The proposition may be rewritten as follows: 


At 
X, = mi = ———_ 3.149 
PI l min{X1, X9}] as + re ( ) 
Moreover, we can generalize the result: let X1,... , X, be independent random 
variables, where X; ~ Exp(Ax), for all k. We have 
At 
P = min{X},... = 3.150 
ie ma Ma ses And] Mach An e190) 


To prove this formula, we can make use of the following proposition. 


Proposition 3.3.5. Let X; ~ Exp(\i), ..., Xn ~ Exp(An) be independent 
random variables, and let Y := min{Xj,...,Xn}. The r.v. Y has an expo- 
nential distribution with parameter \:= A, +...+An- 


Proof. Since min{X,, X2, X3} = min {X, min{X2, X3}}, it suffices to prove 
the result for n = 2. We have, for y > 0, 
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PIY > y) = P[X >y,Xo>y) P(X, > y|P[X2 > y) 
= eT Ye A2¥ — e-OitA2)y 
=> fy(y) = i + Age 2) uly) (3.151) 


Example 3.3.2. Suppose that X,, X2, and X3 are independent random vari- 
ables, all of which have an Exp(A) distribution. Calculate the probability 
P(X, < (Xo + X3)/2]. 


Solution. We have P[X1 < (X2 + X3)/2] = P[Y < X24+ X3], where Y := 
2X. Moreover, for y > 0, 


PIY <y)=P[X1 <y/2}=1-e0/2 = Y~ Exp(\/2) 
Then, by the memoryless property of the exponential distribution, 
P(2X, < Xq+ X3] = 1-—PlY > Xo+ Xz] 
=1-PlY > X2.+X3|Y > Xe|P{Y > X9] 
2 
id. A 
=1-P|Y > X3|P[Y > Xe] =1-| —> 
A+ § 
= 1— (2/3)? = 5/9 
Finally, we would like to have a two-dimensional version of the exponential 
distribution. We can, of course, define 
Fx,,X_(@1, £2) = AyAge 11422) for 2, > 0, ze > 0 (3.152) 


However, the random variables X, and X>2 are then independent. To ob- 
tain a nontrivial generalization of the exponential distribution to the two- 
dimensional case, we can write 


PIX, > z1,X2 > £2] = exp{—A121 — A2X2 — Aj2 max{21, 2}} (3.153) 


for x1 > 0, x2 > 0, where Aj2 is a positive constant. We indeed find that X; ~ 
Exp(A,), for i = 1,2. ~ 


Another possibility is the random vector whose joint density function is 
(see Ref. [5]) 


Ay A2 AZ, + Agxe 2(prA,A2a122)!/? 
Sp oe i ee 
- l-p l-p 
for 1 > 0, x2 > 0, where p € [0,1) is the correlation coefficient of X; and X2, 


and Ip(-) is a modified Bessel’ function of the first kind (of order 0) defined 
by (see p. 375 of Ref. [1]) 


7 Friedrich Wilhelm Bessel, 1784-1846, was born in Germany and died in Kénigs- 
berg, in Prussia (now Kaliningrad, in Russia). He was an astronomer and mathe- 
matician. The mathematical functions that he introduced in 1817 are important 
in applied mathematics, in physics, and in engineering. 


fx,,X,(@1,22) = (3.154) 
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22 1,2)2 1 ,2)3 
G2? , Ge) 


(I)? 2!)? (3!)2 atest (3.155) 


Here, too, we find that X; ~ Exp(A1) and X2q ~ Exp(Ag2). 


Gamma distribution 


Definition 3.3.3 (Reminder). We say that the continuous and nonnegative 
random variable X has a gamma distribution with parameters a > 0 and 
A> 0, and we write that X ~ Gla,X), if 


(Ax)?! \e74* 


Ray) (3.156) 


fx(x) = 
where I'(-) is the gamma function, defined (for a > 0) by 
ioe} 
I'(a) =) te 1e~* dt (3.157) 
0 


Remarks. i) The function (a) is strictly positive for any positive a. For 
a> 1, we have 


oO 
I'(a) = ei a +(a-  f goer af 
0 
°2' 04+ (a—1)P(a—1) =(a—1)IF(a-1) (3.158) 
Then, since 
oO 
T(1) -{ et dt=1 (3.159) 
) 
we have 


P(n) = (n — 1)I'(n — 1) = (n — 1)(n — 2)P'(n — 2) 
=... =(n—1)(n—2)---1-P(1) =(n—1)! (3.160) 


Thus, the gamma function generalizes the factorial function. We also have 
0° - foe) 
I(1/2) = i; #12 e-* dt ale | V2e~*'/? ds (3.161) 
0 0 
ava |” ae" /2 dg = 2 zPIN(0, 1) > 0] = VE 
=< vil e s= is ’ pats = 
0 V20 


ii) Contrary to the exponential distribution, whose density function always has 
the same form, the shape of the density function fx changes with each value of 
the parameter a, which makes it a very useful model for the applications. We 
say that the parameter a is a shape parameter, while \ is a scale parameter (see 
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Fig. 3.1. Examples of probability density functions of G(a, \ = 1) random variables. 


Fig. 3.1). In reality, the shape of the function fx varies mostly when a is small. 
When a increases, the gamma distribution tends to a Gaussian distribution, 
which follows from the fact that if X ~ G(n,A), then the random variable 
X can be represented as the sum of n independent r.v.s X, ~ Exp(A), for 
k =1,...,n (see Prop. 3.3.6). 

iii) If X ~ G(a = 1, A), we have 


fx(z) = Ae7** u(z) (3.162) 


Thus, the gamma distribution generalizes the exponential distribution, since 
G(a = 1,A) = Exp()). 

iv) The parameter a may take any real positive value. When a = n € N, 
the gamma distribution is also named the Erlang® distribution. Moreover, we 
have that G(a = n/2,’ = 1/2) = x2. That is, the chi-square distribution 
with n degrees of freedom, which is very important in statistics, is a particular 
case of the gamma distribution, too. 


The moment-generating function of X ~ G(a, A) is 


re Ae (Ar) 4 A* in 
Mx(t — Np (t-A)a po lg 
( ) [ I (a) I (a) i . 


y=(cte Ae i Sieg * 
Pay(A—t)* Jo P(a)(A— t) 


i Agner Krarup Erlang, 1878-1929, was born and died in Denmark. He was first 
educated by his father, who was a schoolmaster. He studied mathematics and nat- 
ural sciences at the University of Copenhagen and taught in schools for several 
years. After meeting the chief engineer for the Copenhagen telephone company, 
he joined this company in 1908. He then started to apply his knowledge of proba- 
bility theory to the resolution of problems related to telephone calls. He was also 
interested in mathematical tables. 


(a) 
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dr a 
= ene f 3.163 
(5 = :) ort<A ( ) 
We then calculate 
1 
E{X] = Mx(0) = + and E[X?] = M%(0) = ater) (3.164) 
so that 
a(a + 1) a\2 a 
= -(<) =S 3.165 
Vile a= (5) = o3 (3.165) 


We can transform a G(a,) random variable X into an r.v. having a 
G(a,1) distribution, sometimes called the standard gamma distribution, by 
setting Y = \X. Indeed, we then have 


aun) gece (3.166) 


fy(y) = Ftv) oa Flay) 


The distribution function of Y can be expressed as follows: 


Fy(y) = Hey) for y > 0 (3.167) 
where 7(a, y) is the incomplete gamma function, defined by 
y 
y(a,y) = : 1o~ le"? dt (3.168) 
0 


We have the following formula (see p. 272 of Ref. [1]): 
y(a,y) = a ly*e"4¥M(1, 1+, y) (3.169) 


where M(-,-,-) is a confluent hypergeometric function, defined by (see p. 504 
of Ref. [1]) 


a a(at+1)z2 a(a+1)(a+2) 23 
M(a,b,z) = = ——_— — + = = +... 3.170 
(a,)2)=1+ 52+ Fes a+ t@snesa at og) 
When a = 7 EN, the function (a, y) becomes 
n—-l f, 
(ny) =P(n) {1 —e¥ | (3.171) 
k=0 
which implies that 
n-1 k 
Fy(y) =1- eve for y > 0 (3.172) 
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This formula can be rewritten as follows: 
PIY <yj=1-P(W<n-1]=P[W>nj, whereW ~ Poi(y) (3.173) 


Remarks. i) The formula (3.173) can be obtained by doing the integral 


a t?le-tdt (=(n,y)) (3.174) 
0 


by parts (repeatedly). 


ii) We will see in Section 5.1 that, in the case of a Poisson process (with rate 
A = 1), the random variable Y represents the time needed for n events to 
occur, whereas W is the number of events that occur in the interval [0, y]. In 
other words, the relation between Y and W is expressed as follows: the nth 
event of the Poisson process occurs at the latest at time y if and only if there 
are at least n events in the interval (0, y]. 


Example 3.3.3. Suppose that the duration T (in hours) of a major power fail- 
ure is a random variable having a gamma distribution with parameters a = 
2 and \ = 1/2 (so that the average duration is equal to four hours). What 
is the probability that an arbitrary (major) power failure lasts more than six 
hours? 


Solution. First, we have 
P[T > 6] = P[X > 3], where X ~ G(2,1) 
Then, by using the formula (3.173), we can write that 


P[X > 3] = P{Poi(3) < 1] = e3(14+. 3) =4e7? ~ 0.1991 


Remarks. i) It is important not to forget that T (or X) is a continuous random 
variable, while W is discrete. 


ii) When the parameter a is small, as in this example, we can simply integrate 
by parts: 


3 (2) 
1e,0] 
= —re7*|"° +f edz =3e 3 +e =4e% 
3 
We have seen that the exponential distribution is a particular case of the 


gamma distribution. We will now show that the sum of independent exponen- 
tial random variables, with the same parameter, has a gamma distribution. 
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Proposition 3.3.6. Let X1,...,Xn be independent random variables. If X; 
has an exponential distribution with parameter A, for allt, then 


3 X; ~ G(a =n, d) (3.175) 


i=1 


Proof. Let S := S77, X;. We have 


= I ae (44) fort <A (3.176) 


Since [\/(A—t)]” is the moment-generating function of an r.v. having a G(a = 
n,A) distribution, the result is then obtained by uniqueness. Indeed, only the 
G(a = n, X) distribution has this moment-generating function. 0 


The exponential and gamma distributions are models that are widely used 
in reliability. Another continuous random variable, which also generalizes the 
exponential distribution and which is very commonly used in reliability and 
in many other applications, is the Weibull ° distribution. 


Definition 3.3.4. Let X be a continuous random variable whose probability 
density function is given by 


fx(z) = 82S) ee |- (5 ) Jue _¥) (3.177) 


We say that X has a Weibull distribution with parameters 8 > 0, y € R, 
and 6 >0. 


Remark. The exponential distribution is the particular case where 6 = 1, 
y = 0, and 6 = 1/A. Like the gamma distribution, this distribution has a 
shape parameter, namely @. The parameter y is a position parameter, while 
6 is a scale parameter. When y = 0 and 6 = 1, we have 


fx(z) = Brie? u(x) (3.178) 


This variable is called the standard Weibull distribution. Its distribution func- 
tion is 

Fx(z)=1-e"* forz>0 (3.179) 
° E. H. Wallodi Weibull, 1887-1979, was born in Sweden and died in France. In 


addition to his scientific papers on the distribution that bears his name, he is the 
author of numerous papers on strength of materials, fatigue, and reliability. 
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Because the exponential, gamma, and Weibull distributions are very im- 
portant in reliability, we will end this section with the definition of the failure 
rate of a device, or a system, etc. 


Definition 3.3.5. The failure rate (or hazard rate) of a device, whose 
lifetime X is a continuous and nonnegative random variable, is defined by 


rx(t) = AS, (3.180) 


Remarks. i) We have that rx(t) ~ P[X € (t,t + dt] | X > t]/dt. That is, the 
failure rate rx(t), multiplied by dt, is approximately equal to the probability 
that a device being ¢ time unit(s) old will fail in the interval (t,t + dt], given 
that it is functioning at time t. 


ii) To each function rx(t) there corresponds one and only one distribution 
function Fy (t). Indeed, we have 


__fx(t) — &Fx(t) 


from which we can write (since Fy (0) = 0) that 


t t a 
[ rx(s)as f ee as —In{1 — Fx(s)]|p = —Infl — Fx (t)] 


=> Fy(t)=1-exp {-[ rx(s) as} (3.182) 


Particular cases 
1) If X ~ Exp(A), then 


rx(t) =, =A Vt20 (3.183) 


Note that this result is a consequence of the memoryless property of the expo- 
nential distribution. We say that the parameter A is the rate of the exponential 
distribution. 


2) In the case of the standard gamma distribution, we may write that 
ta-le-t 


xO = Taya et eee 


3) Finally, if X has a standard Weibull distribution, we deduce from Eqs. 
(3.178) and (3.179) that 
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rx(t) = 6t?-} (3.185) 


In practice, the function rx(t) has more or less the shape of a bathtub. 
That is, at the beginning the failure rate decreases, then this rate is rather 
constant, and finally the failure rate increases (which is equally true for the 
death rate of humans). To obtain this kind of curve, we can consider a random 
variable that is a linear combination of three Weibull distributions: 


X := ec; X, +9 Xo +0¢3 X3 (3.186) 


with c; > 0, for alli, and c, +c2+¢3 = 1 (called a mized Weibull distribution), 
where X, has a 8 parameter smaller than 1, Xo a 3 parameter equal to 1, 
and X3 a 3 parameter greater than 1. 


3.3.2 Continuous-time Markov chains 


Let {X(t),¢ > 0} be a continuous-time and discrete-state Markovian [see Eq. 
(3.1)] stochastic process, and let 7; be the time that the process spends in 
state 7 before making a transition to some other state. We may write that 


Ply >stt|y>t]=Pl[r>s] Vs,t20 (3.187) 


Indeed, since the process is Markovian, the time it has spent in a given state 
does not influence the future. Consequently, whatever the time that the pro- 
cess has already spent in i, it is as likely that it will remain there during at 
least s additional time units than if it had just entered this state. Eq. (3.187) 
means that the continuous random variable 7; possesses the memoryless prop- 
erty. As we already mentioned (see p. 110), only the exponential distribution 
possesses this property. We can therefore conclude that 7; has an exponential 
distribution, with parameter denoted by 14%;, which, in general, depends on 
state 7. 

Moreover, the Markov property also implies that the next state visited, 7, 
is independent of 7;. Thus, when the process leaves state 2, it enters state 7 
(# 7) with probability p;; (by definition), where 


Pi =O Vi and So pis a Na (3.188) 
j=0 


The p;,;’s are the one-step transition probabilities of the embedded (or as- 
sociated) discrete-time Markov chain. Note, however, that, contrary to the 
transition matrices in the preceding section, all the terms on the main (de- 
creasing from top left to bottom right) diagonal of the matrix are necessarily 
equal to zero, by definition of the p; ;’s in the present case. 

The process {X(t),¢ > 0} is called a continuous-time Markov chain, which 
is now defined formally. 
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Definition 3.3.6. Let {X(t),t > 0} be a continuous-time stochastic pro- 
cess whose state space is N° = {0,1,...}. We say that {X(t),t > 0} is a 
continuous-time Markov chain if 


P[X(t +s) =j | X(s) =i, X(r) =2,,0<r<s] 
= PIX(t+s) =5|X(s) =] =piglt) (3.189) 


V s,t >0 andV i,j,2, € N°. 


Remarks. i) As in the case of the discrete-time Markov chains, we assume that 
the chains considered have stationary or time-homogeneous transition proba- 
bilities. We could treat the general case and denote the conditional probability 
P[X(t) = j | X(s) = 4] by p;,;(s,t), where t > s, but since the most important 
processes for the applications are indeed such that p;,;(s,t) = pi,j(t — s), it 
will not be necessary. 


ii) Continuous-time Markov chains are also known as Markov jump processes. 


ili) The function p;,;(t) is called the transition function of the continuous-time 
Markov chain. 


iv) The probabilities p; ;(¢) correspond to the p's in discrete-time Markov 


chains. If there exist a t > 0 for which p;,;(t) > 0 and a t* > 0 for which 
p;(t*) > 0, we say that states 1 and 7 communicate. The chain is irreducible 
if all states communicate. 


v) We may write that 
So pig(t)=1 Vi (3.190) 
j=0 


since the process must be in some state at time t + s, regardless of the state 
it was in at time s. 


vi) For the sake of simplicity, we will assume in the sequel that the state space 
of the Markov chain {X(t),t > 0} is, save indication to the contrary, the set 
{0,1,...}. However, as in the discrete case, the state space can actually be a 
set Sx (4) C {0,1,...}, or, more generally, a finite or countably infinite set of 
real numbers. 


Example 3.3.4. Since all the elements of the main diagonal of the matrix P 
in Example 3.2.8 are equal to zero, we can consider this matrix as the transi- 
tion matrix of the embedded discrete-time Markov chain of a continuous-time 
Markov chain. The fact that p;,; = 1/2, for all i € 7, does not mean that all 
the random variables 7; have the same parameter v;, for 7 = 0,1, 2. 


The following proposition is the equivalent, in the continuous case, of 
Proposition 3.2.1. 
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Proposition 3.3.7 (Chapman—Kolmogorov equations (2)). For all s,t 
nonnegative, we have 


Di,k( (t+ 8) = Last )P5,n(8 = Ye )p;,K(t (3.191) 


Proof. The proof is similar to that of Proposition 3.2.1. 0 
Example 3.3.5. In general, it is not easy to calculate the functions p;,;(t) ex- 
plicitly. Theorems 3.3.1 and 3.3.2 provide differential equations whose solu- 
tions are these p;,;(t)’s. In the case of a Poisson process (see Section 5.1), we 
find directly that 

At)i-t 

a ee for 7 -—i>0 
Pig(t) = oe 
0 for 7-1 <0 


where A is the rate of the process. We indeed have, for k > 1, 


wag ty, (A5)P3 
Yast )p;,K(8 = Yonsl )pj,4(s ae ceri saree) 


j=i 
k joi gk—Jj Krt aj gk~i-J 
= e~Atts) yk-i ve =e Mtts) yk=i $ 
=e — =e 
2 Gea 2 Fe -7 Hy 
k-i ee re : 
33 - k—i\ tsk-*5 (4) (t + s)*-# 
= Mitts) \k 4 Sal A(t+s ) ko 4 
° > ( j ea . (ki) 
= Att s 3 
= Mien AE IPT 7. ” i = piz(t + 8) 


where Eq. (*) is obtained by Newton’s!? binomial theorem. 


Notation. We denote by p;(t) the (marginal) probability that the process 
{X(t),t > 0} will be in state 7 at time t: 


p;(t) = P[X(t) = J] (3.192) 


If a; := P|X(0) =], fori =0,1,..., then we may write that 
oO 
=> arpi3(t) (3.193) 
i=0 


When P[X(0) = k] = 1 for some k € {0,1,...}, we simply have that p,(t) = 

Pk, g(t). 

10 Sir Isaac Newton, 1643-1727, was born and died in England. Newton was a scholar 
who is famous for his contributions to the fields of mechanics, optics, and astron- 


omy. He is one of the inventors of differential calculus. He also wrote theological 
books. 
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3.3.3 Calculation of the transition function p,,;(t) 


First, if state i is absorbing, which means that the parameter 1; of the random 
variable 7; is equal to 0, we may write that 


lifi=j 
Big (t) = 655 = \ fie ; (3.194) 


for all ¢ > 0. In the case of nonabsorbing states, we will obtain two systems 
of differential equations, which, when we can solve them, give us the value of 
pi,j(t), for all states 7,7 € {0,1,...}. 

Remark. It can be shown that the p,;(t)’s are continuous functions of t, for 
every pair (i, 7). 


Definition 3.3.7. The quantities 
Yj = 4, Di,j Vi#j € {OTs act (3.195) 


are the infinitesimal parameters or instantaneous transition rates of 
the continuous-time Markov chain {X(t),t > O}. 


Remark. We have 


> W445 =U; So Pig =i (because Pit = 0) (3.196) 
j#i xt 
We set 
Wig = (3.197) 
It follows that 
S45 S00 (3.198) 
j=0 


Definition 3.3.8. The matriz 


0 V0.0 Yo,1 Vo,2--- 
1] Yt0 1,1 41,2 --- 


= 2] V2.0 V2,1 2,2 --- (3.199) 


is known as the generating matrix of the continuous-time Markov chain 
{X(t), t 2 O}. 


Remarks. i) If we know the quantities ;,;, for all 1 4 j, then we can calculate 
the rates 4; and the probabilities p;,; from Eq. (3.195). Moreover, we will show 
in this section that the p;,;(¢)’s depend only on the ™’s and the p;,;’s, from 
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which one derives the name of generator of the Markov chain for the set of all 
i,j 8. 

ii) The matrix G corresponds to the transition matrix P for a discrete-time 
Markov chain. However, note that the 4;,;’s are not probabilities and that the 
sum of the elements of each row of G is equal to 0 rather than to 1. 


Notation. If the function g(r) is such that 
tim 2 =o (3.200) 
z-0 


then we write that g(x) = o(z). 


A function g(x) that “is” o(x) must therefore tend to 0 more rapidly than 
the identity function f(a) = « when x > 0. Thus, g;(x) := z? is o(x), while 
g2(x) := \/z is not o(x). Moreover, if g;(x) = o(x), fori =1,... ,n, then 


San =o(r) VGeR (3.201) 
i=1 


Proposition 3.3.8. The probability that a continuous-time Markov chain, 


{X(t),t > 0}, makes two or more transitions in an interval of length 6 is 
o(6). 


Proof. We know that the time 7; that the process spends in state 7 has an 
exponential distribution with parameter v;, for 1 = 0,1,.... Suppose first 
that v; = v V i and, without loss of generality, that X(0) = 0. Let N be the 
number of transitions of the Markov chain in the interval {0, 5]. We have 


P(N >2]=1-—P(N =0|-P[N=]] 


Co Ad 
=1- Pl >8)- >> | folu)poaPlrn > 5~ ul du 
k=179 


0 6 
= ie PO Yon | ve’ eV dy 
k=1 0 


CO 
=1-e —S pope v5 =1-e “(1 +8) (3.202) 


k=1 
Using the series expansion 
Pe) 
ealrets +...=1t+2+ oz) (3.203) 


we may write that 


P[N > 2] =1-—e7(14+ 1d) =1—[1 — vd + 0(4)|(1 +5) = 0(6) (3.204) 
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Now, if the v,;’s are not all equal, it suffices to replace each % by v := 
max{vp,1,...} to prove the result. Indeed, since E[7;] = 1/1, the larger 1; 
is, the shorter the average time that the process spends in state 7 is. But, since 
for the largest v; we can assert that P[N > 2] = o(d), this must hold true 
when vy, <vVi.B 


Proposition 3.3.9. We may write that 
d nots 
Yjg= is (t) Vige {0, Tse } (3.205) 
t t=0 


Proof. Suppose first that i = 7. We know, by the preceding proposition, that 
the probability of two or more transitions in an interval of length 6 is 0(6). It 
follows that 


pis(5) = Plt; > 5] + 0(5) =e” + 0(5) = 1 — 6 + 0(6) (3.206) 
Since p;,;(0) = 1, we may write that 


paa(8) ~ Pas(0) = — 148 + 0(6) ap PbO) —Pusl) Ly, 4 20) (3.207) 


Taking the limit on both sides as 6 decreases to 0, we obtain 
p;,;(0) = YH MH (3.208) 
When i ¥ j, we have that p;,;(0) = 0. Then 


piz(O) = Pir < 6] pij +0(6) = (1- e789) yy + 0(5) = %6pi,3 + 0(6) 
(3.209) 


so that 


Pi,j (9) = pig (0) 


mC) 
3 = % Pig + ie 2 VYiPig =Vig O (3.210) 


, = 
Pj,;(0) = at 5 


We now give the first system of differential equations that enables us to 
calculate the p;,;(t)’s. 
Theorem 3.3.1 (Kolmogorov backward equations). For all states i,j 
€ N°, and for allt > 0, we have 


P(t) = SS Yi,k Pkg (t) (3.211) 
k=0 


Proof. We decompose the probability p;,;(¢) into two incompatible cases: 


pig (t) = Pl; < t, X(t) =5| X(0) =i) + Pln >t, X(t) =5 | X(0) =7 
(3.212) 
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Since P[7; > t] = e~”t, we may write that 
Pir > t, X(t) = 7 | X(0) = i] =e7"*6,; (3.213) 
Next, we have 
t 
Pash x= ol xX) Hs) = / ne (Sodus Pk, g(t — s))as (3.214) 
0 ki 
so that 
t 
Pi,g(t) =e vt 543 + / ye vie ( S- Pik Pk, j (t — s))as (3.215) 
0 ki 


Remark. This integral equation allows us to state that the p;_;(t)’s are continu- 
ous functions of t, as we already mentioned above. It follows that the function 
that we integrate is a continuous function as well. 


Finally, to obtain the Kolmogorov backward equations, it suffices to first 
rewrite Eq. (3.215) as follows: 


t 
Pig (t) = Sig + nen | eo ( Pix prs(w) Ja ee 
: ki 
and then to differentiate this last equation: 


t 
P(t) = —n feb ner f (Sp pus(u) Ja} 
0 


ki 
a ye vit ervit > Pik Pkg (t) 


ki 
= -1 pig(t) + V4 >~ Pik Pkg (t) = Vi svig(t) + >> Mi,k Pkg (t) 
ky ki 
CO 
=S vcpeglt) O (3.217) 
k=0 


Remark. If we set t = 0 in the Kolmogorov backward equations, we obtain 
[o@} 
Pi,j(0) = > ¥i,k Pkg (0) = ¥,9 3,90) = Mig (3.218) 
k=0 


which confirms the result in Proposition 3.3.9. Actually, from Proposition 
3.3.9, we can show directly the validity of the Kolmogorov backward equations. 
Indeed, we deduce from the Chapman—Kolmogorov equations that 
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d cg 
a —pij(s +t) “a” pik (8) Peg(t) Vs,t>0 (3.219) 


=> p(t) = pi p(0) Prg(t) (with s = 0) 
k=0 


co 
= pij(t)= >- vin Pe g(t) 


where we assumed, in the first equation, that we can interchange the derivative 
and the summation. 


The second system of differential equations that we can use to calculate 
the p;,j(t)’s is valid when, as in the remark above, we can interchange the 
derivative and the summation when we differentiate the Chapman—Kolmogo- 
rov equations. Now, this interchange is allowed when the state space of the 
Markov chain is finite and, also, in particular, for the birth and death processes, 
which will be defined in the next subsection. 


Theorem 3.3.2 (Kolmogorov forward equations). Under the condition 
mentioned above, for all i,j € N°, and for any t > 0, we have 


co 
Pi g(t) = S> pin (t) Vij (3.220) 
Proof. We have, under the condition in question, 


d 
qePialt + 8) = Spal < preils s) Vs,t>0 (3.221) 
if s=0 
> pit = Yorual Pi, j (0 


=r Pig (t = Ypial Ve 


by Proposition 3.3.9. 0 


Remarks. i) Since there is a Kolmogorov equation for every pair (i,7), the 
two systems of differential equations comprise m? equations each, where m 
is the number of elements in the state space of the Markov chain { X(t) 
t = 0} (which can be infinite). Thus, obtaining an explicit solution for the 
probabilities p; ;(t) from these systems of equations is generally very difficult. 
In the next subsection, we will calculate the p;,;(t)’s when the state space has 
only two elements. 


ii) In certain particular cases, for example, in the case of the Poisson pro- 
cess, we can determine the p,;(t)’s without having to solve the Kolmogorov 
differential equations. 
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Example 3.3.6. Suppose that the Markov chain {X(¢),t > 0} has state space 
{0,1,...}, and that the process, from state 0, can only move to state 1. That 
is, Po,1 = 1. We then have 


—vjifk=0 
Yok = Yo ifk=1 
0 otherwise 


It follows that the Kolmogorov backward equation for the pair (0,7) is given 
by 
Po,j(t) = volpij(t) — po,5(t)] 


If state 0 is absorbing instead, we have the following trivial result: 
poo(t)=1 and po j(t)=0 for any 7 #0 


Note that since the two functions are constant, the differential equation above, 
which becomes pp ;(t) = 0 (since vp = 0), is satisfied. Conversely, the solution 
of the differential equation po ;(t) = 0 is po,;(t) = c, and the fact that po,;(0) = 
d9;, that is, the initial condition, enables us to determine the value of the 
constant c. 


3.3.4 Particular processes 
A continuous-time, two-state Markov chain 


The first particular case that we consider is that for which the state space of 
the continuous-time Markov chain {X(t),t > 0} is the set {0,1}. We assume 
that these states are not absorbing, so that they communicate. Since the 
process, from state 0 (respectively, 1), can only move to state 1 (resp., 0), we 
have that po,1 = pi,o = 1. The generating matrix of the chain is given by 


G= ie re | (3.222) 
VA VY, 


from which we deduce that the four Kolmogorov backward equations are, for 
any t > 0, 


1 
) Q —V po,o(t) + % Pi,o(t) 

(2) 

= -pY ty+y t 
) ‘) 0 Po,1( ) 0 P1,1( ) (3.223) 
) = % po.o(t) — 1 pi.o(é) 

4) 
) ¢ Y po(t) — 4 pi1(t) 


mS 
Ee ES ON Ot 
ah 


‘oO 
NNO 
oo} 


Moreover, we have the following initial conditions: 


p:j(0) = 5;; for i,j = 0,1 (3.224) 
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Since 


po,o(t) + po,1(t) = pio(t) + pia(t) = 1 (3.225) 


it is sufficient to consider Eqs. (1) and (3) of the system (3.223). We deduce 
from these equations that 


Po,o(t) + Pi,o(t) = (1 — Yo) [p0,0(t) — P1,0(t)] (3.226) 
Thus, if v = 1, we have 
Poo(t) + pio(t)=0 => poolt) +pio(t) =¢ (3.227) 


a constant. The initial conditions (3.224) imply that c = 1. Eq. (1) can there- 
fore be rewritten as follows: 


Po,o(t) = —vo{po,o(t) —1+ poo(t)] <=>  po,o(t) + 2v0 po,o(t) = v0 


(3.228) 
Multiplying on both sides by e?”t, we may write that 
d 
7 (7 noo(t)) = 40 e?" (3.229) 
The general solution of this ordinary differential equation is 
e2vot Po,o(t) = som +9 (3.230) 


where co is a constant. Making use of the condition po,o(0) = 1, we find that 
Co = 1/2, so that 


i 
po,o(t) = . (l+e7?”") fort >0 (3.231) 


From this function, we can calculate the other three functions p;,;(t). 


Remark. The general solution of the ordinary differential equation (0.d.e.) 
F'(z) +c F(z) = G(z) (3.232) 


where c is a constant, is (for r > 0) 
F(z)=e"% (Fo) +f e Gy) ay) (3.233) 
0 


When 19 # 1, subtracting Eq. (3) from Eq. (1) in the system (3.223), we 
find that 


Po,o(t) — Pi,o(t) = —(vo + 1) [p0,0(t) — p1,0(t)] (3.234) 
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Using the preceding remark with F(t) := po,o(t) — pio(t) and G(t) = 0, we 
may write that the solution of this o.d.e. is 


Po,o(t) — P1,0(t) = [po,0(0) — pi,o(0)Je~ Mo t4)* = eT otuatt (3.235) 
Substituting into Eq. (1), we obtain 
Po,o(t) = —yoe Wott (3.236) 


from which we deduce that 


7 t 
ys Po,0(8) ds = -| Yoe” Wotui)s as 
2 0 


iz _~ 
<=> po,o(t) = Po,o(0) — ar (1 ag rey) (3.237) 


That is, 


is V1 4) —(vot)t 
t)= + oer Vt>0 3.238 
Po,o(t) FRE Sem > ( ) 


Note that if vo = 1 in this formula, then we retrieve the formula (3.231). 
Next, by using Eq. (3.235), we find that 


t)= t) — ee otuyt — SVE po e Wott VYt>0 
P1,0(t) = po,o(t) — e ie Wek e > 
(3.239) 
Finally, Eq. (3.225) implies that 
po,i(t) = 1 — poo(t) =—2— - Pet! Vt>0 — (3.240) 


Y+vy Yo + Vy 


and 


40) VY} 


en otnyt Yt>O (3.241) 


t)=1—- t)= 
pi,i(t) Pi,o(é) vy ty my +Y 


As in the discrete case, an important problem (which will be treated in the 
next subsection) is that of determining, if it exists, the limiting probability 
lim; oo Pi,j(t). Here, we easily find that 


V) 


<7, fi =Oandi=0,1 

V0 V| 

jim pig (t) = (3,242) 
00 


“0 if; —1andi=0,1 
YotVy, 


Note that the limit exists and does not depend on the initial state 7. 
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Pure birth processes 


Definition 3.3.9. Let {X(t),t > 0} be a continuous-time Markov chain 
whose state space is the set {0,1,...}. If 


Pii4i = 1 fori = 0, 1504 (3.243) 
the process is called a pure birth process. 


Remarks. i) A pure birth process is thus a continuous-time Markov chain for 
which transitions can only be made from an arbitrary state to its right-hand 
neighbor. It is a particular case of the birth and death processes, which will be 
treated further on. 


ii) We can also define a pure birth process by setting 


Sf wifgaitl 
“j= { 0 otherwise (3.244) 


for all i 4 j € {0,1,...}. 


The Kolmogorov forward equations for a pure birth process are the fol- 
lowing: 


Pi i(t) = —v4 pi a(t) (3.245) 
and 
Pi g(t) = 44-1 Pig—1(t) — Yj Pig(t) IEF =i+1i+2,... (3.246) 


Proposition 3.3.10. The function p,,;(t) for a pure birth process is given, 
for anyt > 0 and for alli € {0,1,...}, by pij(t) = 0 if 7 <i, and 


envit ifj=i 


se ‘ 3.247 
Pst) =) ys erust | e’5*p, »_1(s) ds if j >i (3.247) 
0 


Proof. Since the process can move only to the right, we indeed have p; ;(t) = 0 
if 7 <i. 
Next, let F’ be the following event: the process makes no transitions in the 
interval [0, ¢]. Since the process cannot move backward, we may write that 
pii(t) = PIF] = Pir; > t] =e “it (3.248) 


Finally, by (3.233), the solution of Eq. (3.246) is 


t 
pig(t) =e {rs(0) tyr fee rigaals) as} (3.249) 
0 
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and the result follows from the fact that p;,;(0) = 0, for j =*+1,i1+2,....0 


From the formulas (3.247), we can calculate p,,;(t) recursively. For in- 
stance, we have 


£ 
piiti(t) = never | eX'+1%y; i(s) ds 
0 


t t 
= neve | eVitise Mis ds = nerwnnt | elit vids ds 
0 0 


vite “et if vy; = 441 


(3.250) 


il 


Vi ai Steet 
— (et — 1") ify A ig 
Vit — Vj 


Particular cases 


1) Poisson process. This process is obtained by setting vy; = A, for all 7 € 
{0,1,...}. To obtain its transition function p;,;(t), we can use Eqs. (3.247) 
and (3.250), with y; = 434; = A. We have 


X d (At)? vt 
Piigi(t) =Ate** = > pierol(t)= | er se“ ds = —=~e 
0 
(3.251) 
Proceeding recursively, we obtain the following general formula: 
re = (At)* —At ss 2 
Pii¢k(t) = Re fort >0 and k=0,1,... (3.252) 
That is, 
Diitk(t) => P{[Poi(At) = k] (3.253) 


The parameter . is called the rate of the process. 


We can also obtain the preceding formula without making use of the Kol- 
mogorov equations, proceeding as follows: since vy; = A, the random variables 
7; all have an exponential distribution with parameter \. Moreover, they are 
independent, by the Markov property. It follows, by Proposition 3.3.6, that 


S:=T%j+...4+Ti4h-1~ G(k,A) VRE21 (3.254) 
Then, we may write that 


. rat ey i (As)*~* | As p-At=s) g 
Piitk(t) = | fs(s) Plrizk >t — s]ds = ec Ry a 
0 0 ( _ )! 
es Yael 


t k 
= k-1 —_ pAt (At) 3.955 
=e een ds =e “*———. (3.255) 
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(3.248) 6 ty 


for i,k € {0,1,...} (since p; ;(t) and Vt > 0. 


2) Yule!! process. This process, studied by Yule in the framework of the 
theory of evolution, can be interpreted as follows: we consider a population 
whose members cannot die and such that each member, independently of the 
others, gives birth to descendants according to a Poisson process with rate A. 
It follows that 


y,=idA Vi>0 (3.256) 


Indeed, when X(t) = i, the time 7; that the process spends in this state is 
given by 
Th = min{X),... , Xi} (3.257) 


where the random variables X,,... ,.X; are independent and all have an ex- 
ponential distribution with parameter A. By Proposition 3.3.5, we may write 
that 7 ~ Exp(yj = 7A). 

Remark. Note that if X(0) = 0, then the process remains in this state forever. 


We can calculate the transition function of the process in various ways. 
We find that 


pig(t) = (jaijema —e™M It ifp>i21 (3.258) 
To check this result, we use the fact that Eq. (3.246) becomes 
Di g(t) = (F —1L)Apig-1(t) — GAig(t) for j =i+1,14+2,... (3.259) 


Next, from the formula (3.258), we may write that 
Pij-1(t) = i (=:) Dig (t) (3.260) 
Moreover, we calculate [also from (3.258)| 
Big (t) = iA pig(t) + GF — 1)Ae™™ pi 5-1 (2) (3.261) 
We have 
— iAping(t) + J — DA pig-1(t) = GF — LA dig-1(t) — IAP (O) 
<> (f- 1) — e™) pig-alt) = 9 — #) i, (2) (3.262) 


"1 George Udny Yule, 1871-1951, was born in Scotland and died in England. He 
first obtained an engineering degree. Next, he became interested in the field of 
statistics and wrote important papers on regression and correlation theory. His 
book Introduction to the Theory of Statistics, published for the first time in 1911, 
was very successful. 
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which holds true by Eq. (3.260). 


Finally, the function given in (3.258) does indeed satisfy the initial condi- 
tion 


pig (0) = di5 (3.263) 
Remark. When i = 1, the formula (3.258) becomes 
ngt)=e M1 -e po! fg >] (3.264) 
We can then write that 
piz(t) = P[Geom(p := e™™*) = jf] iff >1 (3.265) 


Now, when X(0) =7 > 1, it is as if we added i independent random variables, 
X1,...,;, all of which have a geometric distribution with parameter p = 
et. We can show that such a sum S has a negative binomial or Pascal}? 
distribution with parameters i and p, whose probability mass function is given 
by 


P[S=j]= (; 7 :) pi(l—p)* for j=i,it+1,... (3.266) 
—4 


The formula for p;,;(t) thus follows directly from that for p,j(t). 


Birth and death processes 


Definition 3.3.10. [f the instantaneous transition rates v;,; of the continuous- 
time Markov chain {X(t),t > 0} are such that 


%j7=0 iflj-i>1 (3.267) 
the process is said to be a birth and death process. 
Definition 3.3.11. The parameters 
Mi=MYier (fori>0) and pwy:=Vvi-1 (fori >1) (3.268) 


are called, respectively, the birth and death rates of the birth and death 
process. 


12 Blaise Pascal, 1623-1662, was born and died in France. He is one of the founders 
of the theory of probability. He was also interested in geometry and in physics, 
in addition to publishing books on philosophy and on theology. 
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Remark. Other terms used to designate the parameters A; and j; are the fol- 
lowing: growth or arrival rates, and mortality or departure rates, respectively. 
From the instantaneous transition rates 14,;, we can calculate the param- 


eters 1; of the random variables 7;, as well as the probabilities p;,;, for all 
states 7 and 7 in the set {0,1,...}. We have 


Vo = —V0,0 = V0,1 = Ao (3.269) 
and 
W= Vig = Vigan + Vina = A+ omi Ift 21 | (3.270) 
In the case when the state space is the finite set {0,1,... ,m}, we have 
Um = —Umym = Ymym-1 = bem (3.271) 


If A; = py = 0, then state i is absorbing. For all nonabsorbing states 
i € {0,1,...}, we have that po; = 1 and, using the formula (3.195), 


Pit = ATs = Ai =1- DPiji-1 if i 2 1 (3.272) 
VY; N+ Ki 


Remarks. i) A birth and death process is a continuous-time Markov chain for 
which transitions, from state i, can only be made to i — 1 (if > 0) ori +1. 
In a short interval, of length 5, the probability that the process moves from 
state 7 to i+ 1 (respectively, i — 1) is equal to Ajé + 0(6) (resp., pu;d + 0(4)). 
Thus, 1 — (A; + 4;)d + 0(6) is the probability that the process will still be (or 
will be back) in state 7 after 6 unit(s) of time. 

ii) In many applications, the state of the process at a given time instant is 
the number of individuals in the system at this time instant. A birth and 
death process may then be interpreted as follows: when X(t) = 7, the waiting 
time until the next arrival is a random variable X; having an Exp(A;) distri- 
bution and which is independent of the waiting time Y;, having an Exp(,i) 
distribution, until the next departure. We then have t9 = Xo ~ Exp(v = Ao) 
and 


7% = min{X;, Yj} ~ Exp(y, =A; + pi) fori >0 (3.273) 


Moreover, we indeed have pp,i = 1 and 


ind. ri aa ve 
Pits = PIX: < Yi] = PIEXp(\i) < Exp(ui)] "SG fi > 0 (3.274) 
v v 


Similarly, 


by eps 
Pil = ae ifi >0 (3.275) 


4 
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iii) A pure birth process is the particular case where the death rates 4; are 
all equal to zero. A process for which 4; = 0 for all i is called a pure death 
process. 


Particular cases 


1) The continuous-time Markov chain whose state space is the set {0,1}, 
considered at the beginning of this subsection (see p. 129), is an example of a 
birth and death process, for which 


o=Y% and p= (3.276) 


(all the other birth and death rates being equal to zero). 


2) Suppose that we modify the Yule process as follows: after an exponential 
time with parameter A, an individual either gives birth to a descendant, with 
probability p, or disappears from the population, with probability 1 — p. Let 
X(t) be the number of individuals in the population at time t. The process 
{X(t),t > 0} is now a birth and death process whose birth and death rates 
are given by 


Ai =ipX and pw;y=i(l—p)A fori >0 (3.277) 


If we assume rather that, when an individual gives birth to a descendant, 
there is a probability p (respectively, 1 — p) that this individual remains in 
(resp., disappears from) the population, then the process {X(t),t > 0} is no 
longer a continuous-time Markov chain. Indeed, suppose that X(0) = 1. We 
may write that 


Pl <t]= Sopp) P(X +... + Xe Se] (3.278) 
k=1 


where the X;’s are independent random variables that all have an exponen- 
tial distribution with parameter . This result follows from the fact that we 
perform independent trials for which the probability of success, that is, the 
case when the individual remains in the population, is equal to p. Since the 
random variable 7; does not have an exponential distribution, {X(t),t > 0} is 
not a continuous-time Markov chain anymore. Actually, 7 is an infinite linear 
combination of independent gamma distributions. 


3) The queueing model M/M/s. This process will be studied in detail 
in Section 6.3.1. We suppose that customers arrive at a system according 
to a Poisson process with rate \. There are s servers, but the customers 
form a single queue. The service times are independent and have an Exp(.) 
distribution. We have 


M=A Vi>0, we=ip if1<i<s, and pp=sp ifi>s (3.279) 
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(since if all the servers are occupied, then the departure rate from the system 
is equal to sj, for any number of persons in the system). 

Remark. M means that the waiting (and service) times are exponential, thus 
Markovian. 


To end this subsection, we give the Kolmogorov backward and forward 
equations for a birth and death process. First, the backward equations are 


Po,j(t) = —Aopo,j(t) + Ao Pr, (t) (3.280) 
if i 
Ph g(t) 2° us Dir (t) — (Ar + ma) Ping (t) + AvPi4a g(t) (3-281) 
In the case of the Kolmogorov forward equations, we find that 


Pio(t) = pn pit) — Ao pio(t) (3.282) 
if j7>0 
pi j(t)) = Aja Pig—r(t) — (Ag + Hg) Ping (t) + Hy4a Pigea(t) (3-283) 


3.3.5 Limiting probabilities and balance equations 


Since it is generally very difficult to solve explicitly the Kolmogorov equations 
to obtain the transition functions p;,;(t), we must often content ourselves with 
the computation of the limiting probability that the process will be in a given 
state when it is in equilibrium. To obtain these limiting probabilities, we can 
try to solve a system of linear equations called the balance equations of the 
process. 


Definition 3.3.12. Let 7, 7, ... be nonnegative real numbers such that 
oO 
ee! (3.284) 
j=0 


If the equation 
. oo 
SS Ny pi,;(t) = 1; (3.285) 
i=0 


is satisfied for all j € {0,1,...} and for allt > 0, then m := (m0,™,...) 18 @ 
stationary distribution. 
The expression stationary distribution is used because if we assume that 
P{X(0) =j]=7; for all j € {0,1,...} (3.286) 


then we have 


PIX(t) = J] = )\PIX() = 51 X() = J P(X) = 7 
i=0 
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= 2 piglt) a OO” my (3.287) 
i=0 
That is, 
PIX(t) = j) = PIX(0) = j] = 73 (3.288) 


Vje{0,1,...} and Vi>0. 


As in the discrete case, we can define the notion of recurrence of a state. 


Notation. We denote by T;,,; the time elapsed between two consecutive visits 
to state i of the continuous-time Markov chain {X(t),¢ > 0}. 


Definition 3.3.13. We say that state i is recurrent if P[T;,; < oo] =1 and 
transient if P[Ti,; < co] <1. Moreover, let 


bi, = E\T; 3] (3.289) 


The recurrent state i is said to be positive (respectively, null) recurrent if 
Mis < 00 (resp., = 00). 


We can prove the following theorem. 


Theorem 3.3.3. If the continuous-time Markov chain {X(t),t > O} is ir- 
reducible and positive recurrent, then it has a unique stationary distribution 
m =(m0,71,-..), where 7; is the limiting probability 


j= jim pis (t) for all j € {0,1,...} (3.290) 


Remarks. i) Note that the 2;’s do not depend on the initial state 7. The 
quantity 1; is also the proportion of time that the Markov chain spends in 
state 7, on the long run. 


ii) If the Markov chain is transient or null recurrent, then it does not have a 
stationary distribution. 


iii) Contrary to the discrete-time Markov chains, a continuous-time Markov 
chain cannot be periodic. This follows from the fact that the time the pro- 
cess spends in an arbitrary state is a random variable having an exponential 
distribution, which is continuous. Consequently, if the limiting probabilities 
exist, then we say that the Markov chain is ergodic. 


iv) We find that 


= for all j € Sx) = {0,1,...} (3.291) 
YsLj,j 
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where the state space Sx z) may, of course, be finite, but must comprise more 
than one state. The interpretation of this result is the following: vj 1 is the 
average time that the process spends in state j on an arbitrary visit to this 
state, and p4;,; is the average time between two consecutive visits to state j, 
so that t/y;,; is the average number of visits to state j over a period of length 
t, where t is large. Thus, over this long period of time, the proportion of time 
during which the chain is in state 7 is effectively given by the ratio v5 : eee 


v) If state 7 is absorbing, we consider it as positive recurrent. The theorem 
applies even in the case when the state space contains a single state, say 0. 
We then have, trivially, that 7 = 1. 


Now, if we differentiate both sides of Eq. (3.285) with respect to t, we 
obtain 


fo.) 
So mpi g(t) =0 Vie {0,1,...} and Vt>0 (3.292) 
i=0 

With ¢ = 0, we may write, by Proposition 3.3.9, that 


CoO 
So mvj =0 Vie {0,1,...} (3.293) 


i=0 


Remarks. i) If the number of states is finite, we can indeed interchange the 
derivative and the summation. However, when the state space is infinite, this 
interchange, though allowed here, must be justified. 

ii) It can be shown that Eq. (3.293) is satisfied if and only if Eq. (3.285) is 
also satisfied. 


Since v;,; = —v;, we can rewrite Eq. (3.293) as follows: 
mvj =) mM VIE {0,1,...} (3.294) 
tAG 


Definition 3.3.14. The equations above are called the balance equations 
of the stochastic process {X (t),t > O}. 


Remarks. i) We can interpret the balance equations as follows: the rate at 
which the process leaves state j must be equal to the rate at which it enters 
state j, for all 7. Now, the rate at which transitions occur, on the long run, 
from state 7 to any other state j 4 i of the Markov chain is given by 7;%,;, 
since 7; is the proportion of time that the process spends in state 7 during the 
period considered, and 1; is the rate at which, when it is in state i, it enters 
j. Given that the process spends an exponential time with parameter v; in 
state 7, 7;v; is the departure rate from j, and the sum of the terms 7%; 
over all states 1 # 7 is the arrival rate to j. 
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ii) The limiting probabilities 7; can be obtained by solving the system (3.294), 
under the condition }°72o 7j = 1 [see (3.284)]. 

We will now obtain a general formula for the 7;’s when {X(t),t > 0} isa 
birth and death process. First, we give a proposition that tells us when the 
73's do exist. 


Proposition 3.3.11. Let {X(t),t > 0} be an irreducible birth and death pro- 
cess, whose state space is the set {0,1,...}. We set 


cele VO WEED © ia rare 
Si= > ee and 53S Hibas** Mk (3.295) 
rae Mipas++ Uk fay AyAg: ++ Ag 


The Markov chain is 


positive recurrent if S; < oo 
null recurrent if S; = Sg = 00 (3.296) 
transient if Sz < oo 


Remark. If the state space of the irreducible birth and death process is finite, 
then we necessarily have that 5S; < oo, so that the Markov chain is positive 
recurrent. 


With the help of this proposition, we can prove the following theorem. 
Theorem 3.3.4. For an irreducible and positive recurrent birth and death 


process {X(t),t > O}, whose state space is the set {0,1,...}, the limiting 
probabilities are given by 


i; 


=a forj=0,1,... 3.297 
where 
ect, san eI (3.298) 
HMiflgs* fj 


Proof. In the case of a birth and death process, the balance equations become 


state j departure rate from 7 = arrival rate to 7 


AoTo = Ha™ 
1 (Ay + 41) = fete + AoTo 
k (21) (Ak + He) = BetiTep1 + Ak-1Tk=-1 


Adding the equations for 7 = 0 and 7 = 1, then for 7 = 1 and j = 2, etc., we 
find that 
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AjTj = 5417541 Vj >0 (3.299) 
which implies that 
Fg aka (3.300) 
Te pages feof 


Using the fact that )°5°) 7; = 1, we obtain 


-1 
AoAi:* IIo 
3 Hipe: ‘as Drwo Lk 
so that 
0; 
W=—aeo «Cfo g = 0,1,... (3.302) 
; ae Ir 


Finally, it can be shown that 7 = (mp, 71,...) is astationary distribution of 
the Markov chain {X(t),¢ > 0}. By Theorem 3.3.3, this distribution is unique. 
Thus, the limiting probabilities are indeed given by the preceding equations. 

Oo 


Remark. We can rewrite the formula (3.297) as follows: 


I; 
(= = 01 e.4 3.303 
Tj ith for j = 0,1, ( ) 


where Sj is defined in (3.295). 


Example 3.3.7. In the case of the two-state, continuous-time Markov chain 
(see pp. 129 and 137), we have 


r 
9=2=" 
fy 4 
from which we deduce that 
1 Vy 
T — = =l-7 
ee yo +4 : 


Note that these results correspond to those given by Eq. (3.242). 
When vp = 1, we have that 7 = 7 = 1/2, which had to be the case, by 
symmetry. 
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3.4 Exercises 


Section 3.2 


Question no. 1 

We suppose that the probability that a certain machine functions without 
failure today is equal to 
0.7 if the machine functioned without failure yesterday and the day before 
yesterday (state 0), 
0.5 if the machine functioned without failure yesterday, but not the day before 
yesterday (state 1), 
0.4 if the machine functioned without failure the day before yesterday, but 
not yesterday (state 2), 
0.2 if the machine did not function without failure, neither yesterday nor the 
day before yesterday (state 3). 


(a) Find the one-step transition probability matrix of the Markov chain asso- 
ciated with the functioning state of the machine. 

(b) Calculate ee that is, the probability of moving from state 0 to state 1 
in two steps. 

(c) Calculate the average number of days without failure of the machine over 
the next two days, given that the Markov chain is presently in state 0. 


Question no. 2 
Let {X,,n = 0,1,...} be a Markov chain whose state space is the set 
{0,1} and whose one-step transition probability matrix P is given by 


p= i ee 


where 0< p< 1. 
(a) Suppose that p = 1 and that Xo = 0. Calculate E[X9]. 
(b) Suppose that p = 1/2 and that P[Xp9 = 0] = P[Xo = 1] = 1/2. We define 
the continuous-time stochastic process {Y(t),t > 0} by Y(t) = tXjq, for t > 0, 
where [t] denotes the integer part of t. 
(i) Calculate Cy (t,t + 1). 
(ii) Is the stochastic process {Y (t),t > 0} wide-sense stationary? Justify. 
(iii) Calculate limp... P[Xn = 0]. 


Question no. 3 

We consider a Markov chain {X,,n = 0,1,...} having states 0 and 1. 
On each step, the process moves from state 0 to state 1 with probability p € 
(0,1), or from state 1 to state 0 with probability 1 — p. 
(a) Calculate Por: 
(b) Suppose that X9 = 0. Calculate the autocorrelation function Rx (1,13). 
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Question no. 4 
The one-step transition probability matrix P of a Markov chain whose 
state space is {0,1} is given by 


P= er za 


Calculate E[X2] if P[Xo = 0] = 1/3. 


Question no. 5 

Let Y;,¥2,... be an infinite sequence of independent random variables, 
all having a Bernoulli distribution with parameter p = 1/3. We define Xn = 
pan Y;, for n = 1,2,.... Then (see p. 85) the stochastic process {Xn,n = 


1,2,...} is a Markov chain. Calculate ps. 


Question no. 6 
Let {X,,n =0,1,...} be a random walk for which 


2 1 
Pitt = 3 and pii-1 = = for i € {0, +1, £2,...} 


Calculate E[X2 | Xo = OJ. 


Question no. 7 
Let {X,,n = 0,1,...} be a Markov chain whose state space is the set 
{0, 1} and whose one-step transition probability matrix is given by 


01 
P= li] 
(a) Calculate Cx(t1,t2) at t; = 0 and te = 1 if P[Xp = 0] = P[Xo = 1] = 
1/2. 
(b) Find lim, +oo P[Xn =0| Xo = 0]. 


Question no. 8 
Let X1, X2,... be an infinite sequence of independent random variables, 
all having a Poisson distribution with parameter a = 1. We define 


nr 
Vora > Xes in a1, 25s: 
k=1 


Then, {Y,,7 = 1,2,...} is a Markov chain (see p. 85). Calculate Bo that is, 
the probability of moving from state 1 to state 3 in four steps. 


Question no. 9 
The flow of a certain river can be one of the following three states: 


0: low flow 
1: average flow 
2: high flow 
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We suppose that the stochastic process {X,,n = 0,1,...}, where X, repre- 
sents the state of the river flow on the nth day, is a Markov chain. Furthermore, 
we estimate that the probability that the flow moves from state i to state j 
in one day is given by the formula 


a 
Pig = 5 —|t— 35]; 


where 0 < 6; < 1, for i,j = 0,1,2. 
(a) Calculate the probability that the river flow moves from state 0 to state 
1 in one day. 


(b) What is the probability that the river flow moves from state 0 to state 2 
in two days? 


Question no. 10 

A machine is made up of two components that operate independently. 
The lifetime J; (in days) of component 7 has an exponential distribution with 
parameter ;, for i = 1,2. 

Suppose that the two components are placed in parallel and that A; = 
Ag = In2. When the machine breaks down, the two components are replaced 
by new ones at the beginning of the following day. Let X, be the number 
of components that operate at the end of n days. Then the stochastic pro- 
cess {Xn,n = 0,1,...} is a Markov chain. Calculate its one-step transition 
probability matrix. 


Question no. 11 
Let {X,,n = 0,1,...} be a Markov chain whose state space is the set 
{0,1,2,3,4} and whose one-step transition probability matrix is 


1 0 0 00 
0.5 0.2 0.3 0 0 
P=;]0 0 010 
0 0 001 
0 0 1 00 


(a) Calculate the probability that the process will move from state 1 to state 
2 in four steps. 


(b) Suppose that Xp = 1. Let Ny be the number of times that state 1 will be 
visited, including the initial state. Calculate E[Nj]. 


Question no. 12 
A Markov chain {X;,n = 0,1,...} with state space {0,1,2,3} has the 
following one-step transition probability matrix: 


1/21/2 0 0 
1/21/41/4 0 
0 0 1/43/4 
000 1 


P= 
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Assuming that Xo = 1, calculate the probability that state 0 will be visited 
before state 3. 


Question no. 13 
A Markov chain has the following one-step transition probabilities: 


Poo = 1 
Baap Hl pissy dors] 1,2)345%: 


Calculate the probability pt that the chain will move from state 7 to state 0 


for the first time after exactly n transitions, for 7 = 1,2,... 


Question no. 14 


Let 
qp000 


q0p00 
P= |0q0p0 
O00qg0p 
0O00gp 
where p+q = 1 and 0 <p <1, be the one-step transition probability matrix 
of a Markov chain whose state space is the set {0,1, 2,3, 4}. 
(a) Is the chain periodic or aperiodic? Justify. 
(b) Calculate, if they exist, the limiting probabilities 7;. 
Question no. 15 


We perform repeated trials that are not Bernoulli trials (see p. 51). We 
suppose that the probability p, of a success on the nth trial is given by 


1/2 forn=1,2 
Pn = Xntl 
Xn +2 


for n = 3,4,... 


where X, is the total number of successes obtained on the (n — 2)nd and 
(n — 1)st trials. Calculate limn_.oo Dn- 


Question no. 16 

In the gambler’s ruin problem (see p. 101), let Y; be the number of plays 
needed to end the game (with the player being ruined or having reached his 
objective of & units), given that his initial fortune is equal to 7 units, for 
+=0,1,...,k. Show that 


cop ay-2{ FE WI ch ep ey 


El¥;] = 
i(k —i) if p= 1/2 


Indication. We may write that E[Yo] = E[Y;] = 0 and (by conditioning on 
the result of the first play) 
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EY; =1+pEYini)+¢£[¥i-1] for? =1,...,k-1 


Question no. 17 

We consider a branching process for which p; = (1 — p)'p, for? =0,1,..., 
where 0 < p< 1. That is, Y := Z+1, where Z is the number of descendants of 
an arbitrary individual, has a geometric distribution with parameter p. Show 
that the probability go of eventual extinction of the population is given by 


qo =i Pil ~p) itp <1/2 
1 ifp>1/2 


Question no. 18 
Let {X,,n = 0,1,...} be an irreducible and ergodic Markov chain. Sup- 
pose that the chain is in equilibrium, so that we can write that 


P(X, =i] = 7; for all states 7 and for any n 


We can show that the process {X,,k =...,n+1,n,...}, for which the time 
is reversed, is also a Markov chain, whose transition probabilities are given by 
7 . Ty aaa 
dij = P[X, =7 | Xngi =i] = Pi for all states 7,7 
fae 
We say that the chain {X,,n = 0,1,...} is téme-reversible if ai; = pi,j, for 


all i,j. Show that the Markov chain whose state space is the set {0,1,2,3} 
and with one-step transition probability matrix 


1/21/2 0 0 
1/2 0 1/2 0 
0 1/2 0 1/2 
0 0 1/21/2 


P= 


is time-reversible. 


Question no. 19 

In the urn model of P. and T. Ehrenfest for the movement of molecules 
in a gas (see p. 75), suppose that Xo = 1. Let m, := E[X,] be the average 
number of molecules in urn I after n shifts. Show that 


N a \" N 
m=F+(1-x) 0-9) 


Indication. Show first that mya, =1+ {ny My. 


forn =1,2,.... 


Question no. 20 
A Markov chain has the following one-step transition probabilities: po, = 1 
and 
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Dijitl = % (> 0) =1 — Pi,o for i = 1,2,3,... 


(a) Show that all states are recurrent if and only if 
n 
Jim, I] ak = 0 
k=1 
(b) Show that if the chain is recurrent, then all states are positive recurrent 


if and only if 
> II Ak < CO 


Indication. If N is a random variable taking its values in the set N° := 
{0,1,...}, then we have 


E|N] = 5 PIN > ni} 


n=1 


Use this result with the variable N denoting the number of transitions needed 
to return to state 0. 


Question no. 21 
The one-step transition probabilities of a Markov chain whose state space 
is {—3, —2, —1,0,1, 2,3} are given by 


Diit1 = 2/3 for i= —3,-2,... ,2 
Pii-1 =1/3 for i= —2,—-1,...,3 
p-3,3 = 1/3=1—ps,-3 


Remark. This chain can be considered as a particular case of a random walk 
defined on a circle. 


(a) Show that the Markov chain is irreducible. 
(b) Determine the period of the chain. 


(c) Calculate the fraction of time that the process spends, over a long period, 
in state i, for i = —3,... ,3. 


Question no. 22 

(a) Show that in a symmetric random walk {X,,n = 0,1,...}, starting from 
Xo = 0, the probability that state a > 0 will be visited before state —b < 0 is 
equal to b/(a + b). 

(b) In a random walk starting from Xp = 0, what is the probability that there 
will be exactly one visit to state 0? 

(c) In a random walk on {0,1,2,...}, starting from Xo = 0 and for which 


Po,1 = 1, what is the probability that there will be exactly k € {1,2...} 
visit(s) to state 0? 
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Question no. 23 
We consider a branching process {X,,n = 0,1,...} for which Xo = 1 and 


p=e*— fori=0,1,... 
i 


That is, the number of descendants of an arbitrary individual has a Pois- 
son distribution with parameter ». Determine the probability qo of eventual 
extinction of the population if (a) \ = 1n2 and (b) A = In4. 


Question no. 24 
A Markov chain with state space {—2,—1,0,1,2} has the following one- 
step transition probabilities: 


Piiti = 3/4 for i= —2, —1,0, 1 

Pii-t = 1/4 for 4 = —1,0, 12 

p-2,2 =1/4=1- po,» 
(a) Determine the classes of the Markov chain. For each class, establish 
whether it is recurrent or transient and whether it is periodic or aperiodic. 
(b) Do the limiting probabilities exist? If they do, compute them. 
(c) Is the chain time-reversible (see p. 147)? Justify. 
Question no. 25 


We consider an irreducible Markov chain whose state space is the set 
{0, 1,2,3,4} and whose one-step transition probability matrix is given by 


01/32/3 0 0 
00 0 1/43/4 


P=|00 0 1/43/4 
10 00 0 
10 00 0 


(a) What is the period of the chain? 


(b) What is the fraction of time, 7;, that the process spends in state j on the 
long run, for 7 = 0,1, 2,3, 4? 


Question no. 26 

Let (Xn, Yn) be the position of a particle that moves in the plane. We 
suppose that {X,,n = 0,1,...} and {Y,,n = 0,1,...} are two independent 
symmetric random walks such that Xo = Yo = 0. We define D, = \/X?2 + Y7?. 
That is, D, represents the distance of the particle from the origin after n 
transitions. Show that if n is large, then 


P[Dn <di~1l—e?/2” ford >0 


Indication. Express X,, (and Y,,) in terms of a binomial distribution, and use 
the fact that if Z;,Zo,...,Z, are independent random variables having a 
standard Gaussian distribution, then 
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Zot De ti LE OG (oe k/2,) = 1/2) 


Question no. 27 

Let Y,,¥o,... be an infinite sequence of independent random variables, 
all distributed as the discrete r.v. Y whose set of possible values is the set 
Z := {0,+1,+2,...} of all integers. We know (see p. 85) that the stochastic 
process {X,,n = 1,2,...} defined by 


X,=¥,+Yot...+Y, forn=1,2,... 


is a Markov chain and that p;,; = aj;-:, where aj := PlY = ij, for any 7. 
We suppose that |E[Y]| < oo and V[Y] < oo. Show that the chain is 
transient if E[Y] 4 0. 


Question no. 28 
We consider a Markov chain with transition probabilities given by 


1 


_ k+l “A _ 
Pko = E49 a Pkaktl = F9 


fork =0,1,2,.... 

(a) Show that the chain is irreducible and ergodic. 
Indication. See Question no. 20. 

(b) Calculate the limiting probabilities 7,, for k = 0,1,2,.... 
Question no. 29 


The one-step transition probabilities of an irreducible Markov chain are 
given by 


1 
= re — er ace d =o 
P0,1 Pk,O k+l an Pk,k+1 k+l 


for k =1,2,.... 
(a) Calculate Bee for k= 1,2,.... 


(b) Calculate Byes for k = 2 and 3. What is the period of the chain? 

(c) Is the Markov chain transient, positive recurrent, or null recurrent? Justify. 
Indication. See Question no. 20. 

Question no. 30 


We consider a Markov chain defined by the one-step transition probability 
matrix 
1/2 1/4 1/4 
P=|atl-a 0 
0 a tl-a 


where 0 < a < 1. The state space is the set {0, 1,2}. 


(a) For what values of the constant a is the Markov chain irreducible and 
ergodic? Justify. 
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(b) Calculate the limiting probabilities for the values of a found in part (a). 


Question no. 31 

A system is made up of two identical components in standby redundancy. 
That is, only one component is active at a time, and the other is in standby. 
We assume that the lifetime of each component has an exponential distri- 
bution and that the probability that the active component will fail during a 
given day is equal to 0.1. The other component, if it is not also down, then 
relieves the failed one at the beginning of the following day. There is a single 
technician who repairs the components. Moreover, he only starts repairing a 
failed component at the beginning of his workday, and he needs two working 
days to complete a repair. 

Let X;, be the condition of the components at the end of the nth day. The 
process {Xpn,n = 0,1,2,...} is a Markov chain having the following states: 


0: neither component is down 

1: a single component is down and it will be repaired one day 
from now 

2: a single component is down and it will be repaired two days 
from now 

3: both components are down (and a component will be repaired 
one day from now) 


(a) What is the matrix P of one-step transition probabilities of the chain? 


(b) For each class of the chain, determine whether it is transient or recurrent. 
Justify. 

(c) Suppose that Xo = 0. Calculate the probability that there will be a single 
component that is not down at the end of the second day. 


Question no, 32 


Let 
1/3 01/3 1/3 
100 0 
Bs 010 0 
010 O 


be the one-step transition probability matrix of an irreducible Markov chain 
whose state space is the set {0,1,2,3}. 

(a) What is the period of state 1? Justify. 

(b) Calculate, if they exist, the quantities 7;, where 7; is the proportion 
of time that the process spends in state j, over a long period of time, for 
j =0,1,2,3. 

(c) Suppose that the elements of the first row of the matrix P are the p,’s 


of a branching process. Calculate the probability that the population will die 
out if (i) Xo = 2, (ii) Xy <1 (and Xo = 2). 
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Question no. 33 

John plays independent repetitions of the following game: he tosses two 
fair dice simultaneously. If he gets a sum of 7 or 11 (respectively, 2, 3, or 12), 
he wins (resp., loses) $1. Otherwise, he neither wins nor loses anything. John 
has an initial fortune of $i, where 7 = 1 or 2, and he will stop playing when 
either he goes broke or his fortune reaches $3. Let X,, be John’s fortune after 
n repetitions, forn =0,1,.... Then {X,,n = 0,1,...} is a Markov chain. 
(a) Find the matrix P of one-step transition probabilities of the Markov chain. 
(b) For each class of the chain, specify whether it is recurrent or transient. 
Justify. 
(c) Calculate the mathematical expectation of X, if the initial distribution of 
the chain is a9 = 0, a; = 1/2, ag = 1/2, and a3 = 0. 


Question no. 34 
A Markov chain whose state space is the set {0,1,2} has the following 
one-step transition probability matrix: 


0 10 
P={1-p0p 
0 10 


where 0 < p< 1. 
(a) Calculate P(™), for n > 2. 
(b) Find the period of every state of the Markov chain. 
(c) (i) Calculate the proportion of time that the process spends, on the long 
run, in state 0. 

(ii) Is this proportion equal to the limit limp—oo po? Justify. 
Question no. 35 

A machine is made up of two components placed in parallel and that oper- 
ate independently of each other. The lifetime (in months) of each component 
has an exponential distribution with mean equal to two months. When the 
machine breaks down, the two components are replaced at the beginning of the 
next month. Let X;, n = 0,1,..., be the number of components functioning 
after n months. 
(a) Justify why the stochastic process {X,,n = 0,1,...} is a Markov chain. 
(b) Calculate the matrix P of one-step transition probabilities of the Markov 
chain. 
(c) Identify each class of the chain as recurrent or transient. Justify. 
(d) Calculate V[X,] if the two components are functioning at time n = 0. 
Question no. 36 


A Markov chain whose state space is the set {0,1,...} has the following 
one-step transition probabilities: 
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fio = 2/3 and py, =(1/4)* fori=0,1,... and k=1,2,... 


(a) Show that the limiting probabilities 7; exist, and calculate them. 


(b) Suppose that the p;;,’s are the p,’s, k > 0, of a branching process. Calcu- 
late the probability of eventual extinction of the population if Xo = 1. 


Question no. 37 
Let X1,X2,... be an infinite sequence of independent random variables, 
all having a Bernoulli distribution with parameter p € (0,1). We define 


Ms: 
= 


y= forn =1,2,... 


k=1 
Then the stochastic process {Y;,,n = 1,2,...} is a Markov chain (see p. 85). 


(a) Calculate the one-step transition probability matrix P of the Markov 
chain. 


(b) For every class of the chain, find out whether it is recurrent or transient. 
Justify. 

(c) Calculate V[Y,, |X) = 1). 

(d) Let T; := min{n > 1: Y, = 1}. What is the distribution of the random 
variable T;? 


Question no. 38 


Let 
0 1 0 0 
p- 1/2 0 1/2 0 
Tid. 0: gd 200 192 
0 0 1 0 


be the one-step transition probability matrix of a Markov chain {X;,n = 
0,1,...} whose state space is the set {0,1, 2,3}. 

(a) Let 7; be the proportion of time that the process spends in state j, over 
a long period of time. Show that the 7,’s exist and calculate them. 

(b) Calculate the period of the Markov chain. 

(c) Let Tj := min{n > 0: X, = j}. Calculate P[Ty < T3 | Xo = 1]. That 
is, calculate the probability that, from state 1, the process will visit state 0 
before state 3. 


Question no. 39 
We define the Markov chain {Y,,,n = 1,2,...} by 


n 
Y= So Xk forn =1,2,... 
k=1 
where X41, Xo,... is an infinite sequence of independent random variables 
having a binomial distribution with parameters n = 2 and p = 1/2 (see p. 85). 
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(a) Calculate the transition matrix of the Markov chain. 


(b) For every class of the chain, establish whether it is recurrent or transient. 
Justify. 
(c) What is the distribution of the random variable Y2? 
(d) Let T) := min{n > 1: Y, = 1}. Calculate 
(i) P[T, =k], for k = 1,2,..., 
(ii) P[T, < oo]. 


Question no. 40 
We consider the particular case of the gambler’s ruin problem (see p. 101) 
for which k = 4. 


(a) (i) Calculate the matrix P if the gambler, on any play, actually has a 
probability equal to 


1/4 of winning the play 
1/4 of losing the play 
1/2 of neither winning nor losing the play 


(ii) Do the probabilities f := limp... P[Xn = j | Xo = 2] exist? If they 
do, calculate these probabilities. 


(b) Suppose that the gambler, on an arbitrary play, bets 


$1 if his fortune is equal to $1 or $3 
$2 if his fortune is equal to $2 


Calculate, under the same assumptions as in (a), the probability that the 
gambler will eventually be ruined if Xj = $3. 


Question no. 41 

We consider a population of constant size, N, composed of individuals of 
type A and of type B. We assume that before reproducing, and disappearing 
from the population, an arbitrary individual of type A (respectively, B) is 
mutated into an individual of type B (resp., A) with probability a (resp., ). 
Moreover, we assume that at the moment of reproduction, each individual in 
the population has a probability p of giving birth to an offspring of type A, 
where p is the mathematical expectation of the proportion of individuals of 
type A in the population after mutation. Finally, we assume that the individ- 
uals are independent from one another. Let X, be the number of individuals 
of type A in the nth generation before mutation. Then {X,,n = 0,1,...} is 
a Markov chain. 


(a) Calculate p (= p,) if X;, =i. 

(b) Calculate p,;;, for i,7 € {0,...,.N}. 

(c) Suppose that a = 0 and @ € (0,1). For each class of the Markov chain, 
determine whether it is recurrent or transient. Justify. 
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(d) Suppose that there are exactly N4 > 0 individuals of type A in the initial 
generation after mutation. Given that the first of these Ny individuals gave 
birth to an offspring of type A, calculate, with p = Na/N, (i) the mean and 
(ii) the variance of Xj. 


Question no. 42 
Suppose that 


1/4.1/2 1/4 
P=;a 01-a 
& O Lag 


is the one-step transition probability matrix of a Markov chain with state 
space {0, 1, 2}. 

(a) For what values of a and @ is the Markov chain irreducible? 

(b) Suppose that a = 8 = 1/2. Compute, if they exist, the limiting probabil- 
ities 7;, for 7 = 0,1, 2. 

(c) Suppose that Xp = 0. Compute the probability that the process will (i) 
visit state 1 before state 2 and (ii) return to (or stay in) state 0 before visiting 
state 2. 


Question no. 43 
Let {X,,n =0,1,...} be a Markov chain with state space {0,2} and with 
one-step transition probability matrix 


ve 


and let {Yn, =0,1,...} be a Markov chain whose state space is {3, 4} and 
whose matrix P is given by 


Assume that the random variables X,, and Y,, are independent, for all n. 
We define Z, = X, + Y;,. We can show that the stochastic process {Z,,n = 
0,1,...} is a Markov chain whose state space is {3, 4,5, 6}. 

(a) Calculate the matrix Pz of the Markov chain {Z,,n = 0,1,... }. 
(b) For every class of the chain {Z,,n = 0,1,...}, determine whether it is 
recurrent or transient. Justify. 


(c) Give the period of every class of the chain {Z,,n = 0,1,...}. 


(d) Calculate the matrix Pp”) and deduce from it the value of the limit limp... 
P|Z, =3| Zo = 3]. 


Question no. 44 
Let 
0 1/2 1/2 
P=|]0 a 1l-a 
B1-p 0 
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be the one-step transition probability matrix of a Markov chain {Xn,n = 
0,1,...} having state space {0, 1,2}. 

(a) For what values of a and is the Markov chain irreducible? 

(b) Suppose that a = 1/2 and 8 = 1/3. Calculate, if they exist, the limiting 
probabilities 7;, for 7 = 0,1, 2. 

(c) Give the values of a and ( for which the limiting probabilities 7; exist 
and are equal to 1/3, for 7 = 0,1, 2. Justify. 


(d) Suppose that a = 1/2, 8 = 1, and Xp = 1. Let No be the number of 
transitions needed for the process to visit state 0 for the first time. Calculate 
E{No]. 


Question no. 45 

A person buys stocks of a certain company at the price of 3 cents per share 
(what is known as a penny stock). The investor decides to sell her shares if 
their value decreases to 1 cent or becomes greater than or equal to 5 cents. 
Let X,, be the value of the shares (for the investor) after n days. We sup- 
pose that {X,,n = 0,1,...} is a Markov chain having as state space the set 
{0,1,... ,7}, and for which rows 3 to 5 (corresponding to states 2,3, 4) of the 
one-step transition probability matrix P are the following: 


1/8 1/4 1/4 1/4 1/8 0 0 0 
1/12 1/7 1/41/21 1/4 1/71/12 0 
0 1/121/7 1/4 1/21 1/4 1/7 1/12 


(a) Give the other rows of the matrix P. 

Indication. Once the investor has sold her shares, or their price went to zero, 
their value (for the investor) does not change anymore. 

(b) For every class of the chain {Z,,n = 0,1,...}, establish whether it is 
recurrent or transient. Justify. 

(c) Calculate E[X; | {X, = 3} U{X, = 4}]. 

(d) What is the probability that, after exactly two days, the investor (i) sells 


his shares with a profit? (ii) sells her shares with a loss? (iii) loses all the 
money she invested? 


Question no. 46 
A Markov chain {X,,n = 0,1,...} with state space {0,1,2} has the fol- 
lowing one-step transition probability matrix: 


1/201/2 


P=; 01 0 
10 0 


Calculate limn—.oo pi, for all 7,7 € {0,1, 2}. 
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Question no. 47 

We consider the gambler’s ruin problem (see p. 101). Suppose that Xo = 
$1 and that p = 1/4. However, if the player wins, he wins $2 and if he loses, 
he loses $1. His objective is to reach at least $4. Calculate the probability that 
he will achieve his objective. 


Question no. 48 

(a) Calculate the probability go of eventual extinction of the population in a 
branching process for which po = 1/4, py = 1/4, and pg = 1/2. 

(b) Suppose that the individuals can only give birth to twins, so that the 
probabilities in (a) become pj = 1/4, p} = 1/4, and pj = 1/2. Can we assert 
that g5 = q%? Justify. 


Question no. 49 
Let {Xn,n = 0,1,.. .} be a symmetric random walk defined on the set 
{0, +1, +2,...} and such that Xo = 0. We set 


Y, =X? forn=0,1,... 
It can be shown that {Y,,n = 0,1,...} is a Markov chain whose state space 


is {0,1,4,9,...}. 
a) Calculate the one-step transition probability matrix of the chain {Y,,n = 


Seen 


1 
b) Is the stochastic process {Y,,n = 0,1,...} a random walk? Justify. 


o—m 


— 


(c) For each class of the chain {Y,,n =0,1,...}, (i) determine whether it is 
transient or recurrent and (ii) find its period. 


Question no. 50 
Let 

1/201/41/4 
010 0O 
ae 100 O 
0 01/43/4 
be the one-step transition probability matrix of a Markov chain {Xn,n = 
) 


0,1,...} whose state space is {0,1,2,3}. Calculate the limit limp—oo pe , for 


all i,j € {0,1,2,3}. 


Question no. 51 

Suppose that Xo = $i in the gambler’s ruin problem (see p. 101) and 
that p # 1/2. Suppose also that if the player loses, then someone lends him 
(only once) $1 and he starts to play again, independently from what occurred 
previously. However, the probability p becomes p/2. His objective is to reach 
$k (without taking into account the dollar that someone may have lent him), 
where k > 7. Calculate the probability that he will achieve his objective. 
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Question no. 52 

(a) Calculate, assuming that Xo = 1, the probability go of eventual extinction 
of the population in a branching process for which p; = 1/4, for ¢ = 0,1, 2, 3. 
(b) Find another distribution of the p;’s for which the value of the probability 
go is the same as that in (a). 


Question no. 53 

A machine is composed of two components. The lifetime T; of component 
i has an exponential distribution with parameter ;, for i = 1,2. When the 
machine breaks down, a technician replaces the failed component(s) at the 
beginning of the next time unit. Let X,, n = 0,1,..., be the number of 
components that are not down after n time unit(s). 
(a) Is the stochastic process {X,,n = 0,1,...} a Markov chain if the compo- 
nents are placed in series? If it is, justify and calculate the one-step transition 
probability matrix P of the chain. If it’s not, justify. 
(b) Suppose that the components are placed in parallel and operate indepen- 
dently from each other, but that only one component is active at a time. That 
is, the components are in standby redundancy. In this case, {X,,n = 0,1,...} 
is a Markov chain. Calculate its transition matrix if Ay = Ao. 


Question no. 54 

Suppose that, in the preceding question, the components are placed in 
parallel and operate (independently from each other) both at the same time. 
We say that they are in active redundancy. 
(a) The stochastic process {X,,n = 0,1,...} isa Markov chain. Calculate its 
matrix P. 


(b) Let 
O if neither component is operating 
oe 1, if only component no. 1 is operating 
n°) 12 if only component no. 2 is operating 
2 if both components are operating 
after n time unit(s), for n = 0,1,.... Is the stochastic process {Y,,n = 


0,1,...} a Markov chain? If it is, justify and calculate the transition matrix 
of the chain. If it’s not, justify. 


Question no. 55 

A system comprises two components placed in parallel and operating (both 
at the same time) independently from each other. Component 7 has an expo- 
nential lifetime with parameter A;, for i = 1,2. Let X,, n = 0,1,..., be the 
number of active components after n time unit(s). 


(a) Suppose that, after each time unit, we replace the failed component(s). 
Calculate the one-step transition probability matrix P of the Markov chain 
{amt Oe lise} 

(b) It can be shown that the limiting probabilities 7; exist in part (a). Cal- 
culate these limiting probabilities. 
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(c) Suppose that A; = Ag and that, after each time unit, we replace a single 
component, only when the system is down. Moreover, assume that Xo = 2. 
(i) Calculate the matrix P of the Markov chain {Xn,7 = 0,1,... }. 
(ii) What is the number of classes of the chain? 


Question no. 56 

We consider a symmetric random walk in two dimensions (see p. 75), whose 
state space is the set {(7, 7): 4 = 0,1,2;7 = 0,1,2}. Moreover, we suppose that 
the boundaries are reflecting. That is, when the process makes a transition 
that would take it outside the region defined by the state space, then it returns 
to the last position it occupied (on the boundary). 


(a) Calculate the one-step transition probability matrix of the Markov chain. 


(b) Show that the limiting probabilities exist and calculate them. 


Question no. 57 

A player has only one money unit and wishes to increase his fortune to five 
units. To do so, he plays independent repetitions of a game that, in case of a 
win, yields double the sum he betted. In case of a loss, he loses his bet. On 
each play, he bets an amount that, if he wins, enables him either to exactly 
reach his objective or to get as close as possible to it (for example, if he has 
three units, then he bets only one). We suppose that the game ends when the 
player either has achieved his target or has been ruined and that he has a 
probability equal to 1/2 of winning an arbitrary play. 


(a) What is the probability that he reaches his target? 
(b) What is the average number of repetitions needed for the game to end? 


Question no. 58 

A particle moves in the plane according to a two-dimensional symmetric 
random walk (see p. 75). That is, the particle has a probability equal to 
1/4 of moving from its current position, (Xn, ¥n), to any of its four nearest 
neighbors. We suppose that the particle is at the origin at time n = 0, so that 
Xo = Yo = 0. Thus, at time n = 1, the particle will be in one of the following 
states: (0,1), (0,—1), (1,0), or (—1,0). Let 


Dia Key? 
be the square of the distance of the particle from the origin at time n. Calculate 
E[DzI. 


Question no. 59 

Show that for a symmetric random walk of dimension k = 2 (see Question 
no. 58), the probability that the number of visits to already visited states will 
be infinite is equal to 1. Generalize this result to the case when k € N. 
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Question no. 60 

A machine is composed of two identical components placed in series. The 
lifetime of a component is a random variable having an exponential distribu- 
tion with parameter py. We have at our disposal a stock of n — 2 new compo- 
nents that we differentiate by numbering them from 3 to n (the components 
already installed bearing the numbers 1 and 2). When the machine fails, we 
immediately replace the component that caused the failure by the new com- 
ponent bearing the smallest number among those in stock. Let T’ be the total 
lifetime of the machine, and let N be the number of the only component that, 
at time T, will not be down. Find (a) the probability mass function of N, (b) 
the mathematical expectation of T, and (c) the distribution of T. 


Question no. 61 

We use k light bulbs to light an outside rink. The person responsible for the 
lighting of the rink does not keep spare light bulbs. Rather, he orders, at the 
beginning of each week, new light bulbs to replace the ones that burned out 
during the preceding week. These light bulbs are delivered the following week. 
Let X, be the number of light bulbs in operation at the beginning of the nth 
week, and let Y,, be the number of light bulbs that will burn out during this 
nth week, for n = 0,1,.... We assume that, given that X, =i, the random 
variable Y,, has a discrete uniform distribution over the set {0,1,... ,7}: 


Pia =5|Xn =i) => for 7 =0,1,...,iandi=0,1,...,k 


(a) Calculate the one-step transition probability matrix of the Markov chain 
{Xn,n =0,1,...}. 

(b) Show that the limiting probabilities of the chain {X,,n = 0,1,...} exist 
and are given by 


2(i +1) 


—_—— i=0,1,...,k 
(k+1)(k +2) py Oih 


y= 


Question no. 62 

Electric impulses are measured by a counter that only indicates the highest 
voltage it has registered up to the present time instant. We assume that the 
electric impulses are uniformly distributed over the set {1,2,... , N}. 


(a) Let X,, be the voltage indicated after n electric impulses. The stochastic 
process {X,,n = 1,2,...} is a Markov chain. Find its one-step transition 
probability matrix. 


(b) Let m,; be the average number of additional impulses needed for the 
counter to register the maximum voltage, N, when the voltage indicated is i, 
fori =1,2,...,N—1. 

(i) Obtain a set of difference equations (as in the absorption problems, 
p. 100) for the m,’s, and solve these equations to determine mj, for all 1. 
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(ii) Calculate directly the value of the m,’s without making use of the 
difference equations. 


Question no. 63 

Let X,, Xo, ... be an infinite sequence of independent and identically 
distributed random variables, such that px,(z) = 1/3 if « = —1,0,1. The 
stochastic process {Y,,,n = 1,2,...} defined by 


n 
Vie a? toa De 


i=l 


is a Markov chain (see p. 85). 


(a) Calculate the one-step transition probability matrix P of the chain. 


(b) Give an exact formula for ps, for k = 1,2,.... 


(c) Use the central limit theorem to obtain an approximate formula for pi) 


when & is large enough. 


Question no. 64 

A player has $900 at his disposal. His objective is reach the amount of 
$1000. To do so, he plays repetitions of a game for which the probability 
that he wins an arbitrary repetition is equal to 9/19, independently from one 
repetition to another. 


(a) Calculate the probability that the player will reach his target if he bets 
$1 per repetition of the game. 
(b) Calculate the probability that the player will reach his target if he adopts 
the following strategy: he bets 


$(1000 — x) if 500 < x < 1000 
$a if0 <a < 500 


where x is the amount of money at his disposal at the moment of betting. 
(c) What is the expected gain with the strategy used in (b)? 


Question no. 65 

We consider a system made up of two components placed in parallel and 
operating independently. The lifetime 7; of component no. 1 has an expo- 
nential distribution with parameter 1, while that of component no. 2 is a 
random variable T, ~ Exp(1/2). When the system fails, 50% of the time the 
two components are replaced by new ones, and 50% of the time only the first 
(of the two components) that failed is replaced. Let X, = 1, (respectively, 
12) if only component no. 1 (resp., no. 2) is replaced at the moment of the 
nth failure, and X,, = 2 if both components are replaced. We can show that 
{Xn,n =1,2,...} is a Markov chain. 


(a) Find the one-step transition probability matrix of the chain. 
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(b) For each class of the chain, specify whether it is transient or recurrent. 
Justify. 


(c) Calculate ee fork =1,2,.... 
(d) What is the period of state 12? Justify. 


Question no. 66 


In the gambler’s ruin problem (see p. 101), suppose that the player has an 
infinite initial fortune and that p = 1/2. Let X, be the gain (or the loss) of 
the player after n repetitions of the game. Suppose also that when X, < 0, 
the player plays double or quits. That is, if X, = —1, then Xn+1 =0 or —2; 
if X, = —2, then X,41 = 0 or —4, etc. The process {Xn,n = 0,1,...} isa 
Markov chain whose state space is the set {... ,-4, -2,~1,0,1,2,...}. 


(a) Find the one-step transition probability matrix of the chain. 

(b) Suppose that the state space is {—4,—2,—1,0,1,2} and that p_4a4 = 
p2,2 = 1/2 [the other probabilities being as in (a)]. Calculate, if they exist, 
the limiting probabilities. 

(c) Suppose now that the player decides to stop playing if his losses or his 
profits reach four units. Calculate the probability that the player will stop 
playing because his losses have reached four units, given that he won the first 
repetition of the game. 


Question no. 67 
The state space of a Markov chain {X,,7 = 0,1,2...} is the set of non- 
negative integers {0,1,2,...}, and its one-step transition probability matrix 
is given by 
0 1 0... 
1/2 0 1/2 0 
P= 1]1/31/3 0 1/3... 
1/41/41/4 0 1/40... 
(a) Calculate pe). 
(b) For each class of the chain, determine whether it is transient or recurrent. 
Justify. 
(c) Find the period of each class. 


(d) Let T;,; be the number of transitions needed for the process to move from 
state i to state 7. Calculate P[Ti,0 < T1,3 | {1,0 < 2} U {Ti,3 < 2}]. 


Question no. 68 
We consider a Markov chain whose state space is the set {0,1,2,...} and 
for which 
Pig =P) >O0 Vizge {0,1,2,...} 


Calculate, assuming they exist, the limiting probabilities of the chain. 
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Question no. 69 


Suppose that, in the gambler’s ruin problem (see p. 101), X» = 1 and 
k = 4. Moreover, suppose that the value of p is not constant, but rather 
increases with X,,. More precisely, the one-step transition probability matrix 
of the chain {X,,n = 0,1,2...} is 


Calculate the probability that the player will achieve his objective. 


Question no. 70 


Let {Xn,nm = 0,1,2...} be a branching process for which Xo = 1. Suppose 
that there are two types of individuals, say A and B. Every individual (of type 
A or B) can give birth (independently from the others) to descendants of type 
A or B according to the formula 


PiN4g =m,Ng =n|=1/9 Vm,ne {0,1,2} 


where Ny (respectively, Ng) is the number of descendants of type A (resp., 
B). 
(a) Calculate E[X, | X, > 0). 


(b) Show that the probability of eventual extinction of the population is 
Go ~ 0.15417. 


Section 3.3 


Question no. 71 

Let Y := min{.X1, X2}, where X, and X2 are two independent exponential 
random variables, with parameters A; and A», respectively. We know (see 
p. 113) that Y ~ Exp(A; + Ao). Find the probability density function of the 
random variable Z := Y | {X1 < X9}. 
Remark. We can express Z as follows: Z := Y | {Y = X,}. However, the 
random variables Y | {Y = Xi} and Xj are not identical, because 


PIY <y|¥ =X] = PIX <y|Y = Xi] = Pik Sy |X < X] 
# PIX < yl] 
Note that the events {X; < y} and {X, < X29} are not independent. 


Question no. 72 
Let X1,...,Xn be independent random variables having an exponential 
distribution with parameter .. 
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(a) Use the memoryless property of the exponential distribution to show that 


1 


1 
PIX, > X2+ Xs] = 7 = Ba 


(b) Show, by mathematical induction, that 


n 

1 

phx > Som = sr forn=2,3,... 
k=2 


(c) From the result in (b), calculate the probability P[2Y > S7f_, Xx], where 
¥ max X14, 068 Xp}: 


Question no. 73 

A birth and death process having parameters A, = 0 and pn = pH, for 
all n > 0, is a pure death process with constant death rate. Find, without 
making use of the Kolmogorov equations, the transition function p;,;(¢) for 
this process. 


Question no. 74 
Let X; ~ Exp(A;) and X2 ~ Exp(A2) be two independent random vari- 
ables. Show that, for all x > 0, 


Mt 
Ay + Ag 


Pixs < Xo | min{X, X2} = z] = PIX, < Xo) = 


Question no. 75 

A system is made up of three components placed in standby redundancy: 
at first, only component no. 1 is active and when it fails, component no. 2 
immediately relieves it. Next, at the moment when component no. 2 fails, 
component no. 3 becomes active at once. When the system breaks down, the 
three components are instantly replaced by new ones. Suppose that the life- 
time 7; of component no. k has an exponential distribution with parameter 
Ag, for k = 1, 2,3, and that the random variables T;, T2, and 73 are indepen- 
dent. Let the number of the component that is active at time ¢ be the state 
of the system at this time instant. Write the Kolmogorov backward equations 
for this system. 


Question no. 76 

A university professor cannot receive more than two students at the same 
time in her office. On the day before an exam, students arrive according to a 
Poisson process with rate \ = 3 per hour to ask questions. The professor helps 
the students one at a time. There is a chair in her office where a person can 
wait his or her turn. However, if a student arrives when two other students 
are already in the professor’s office, then this student must come back later. 
We suppose that the time that the professor takes to answer the questions of 
an arbitrary student is an exponential random variable with mean equal to 15 
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minutes, independently from one student to another. If we consider only the 
days preceding an exam, calculate (with the help of the limiting probabilities) 


(a) the average number of students in the professor’s office, 


(b) the proportion of time, on the long run, when the professor is not busy 
answering questions. 


(c) If the professor spent twice more time, on average, with each student, what 
would be the answer in (b)? 


Question no. 77 

Let {X(t),t > 0} and {Y(t),t > 0} be two continuous-time indepen- 
dent Markov chains. We consider the two-dimensional stochastic process 
{(X(t), Y(t)),t = 0}. Find the parameters v7.4), P(i,&),(j,4)» 20d PEi,k),(é,1) OF 
this process. 


Question no. 78 

We consider a pure birth process for which, when there are n individuals 
in the population, the average time (in hours) needed for a birth to occur is 
equal to 1/n, for n > 0. 


(a) Knowing that at time ¢ there are two individuals in the population and 
that at time t + 1 there are still two, what is the probability that the next 
birth will take place between t +- 2 and t + 3? 

(b) If, at the origin, the population is composed of a single individual, what 
is the probability that there will be exactly four births during the first two 
hours? 


Question no. 79 

A factory has m machines. Each machine fails at an exponential rate p. 
When a machine fails, it remains down during a random time having an 
exponential distribution with parameter 4. Moreover, the machines are in- 
dependent from one another. Let X(t) be the number of machines that are 
in working order at time t > 0. It can be shown that the stochastic process 
{X (t),t > 0} is a birth and death process. 


(a) Find the birth and death rates of the process {X(t),t > O}. 
(b) Show that 


jim P[X(t) =n] = P[B(m aa) =n 


forn = 0,1,...,m. 


Question no. 80 

Let {X(t),t > 0} be a pure birth process such that A; = jA, for j = 
0,1,..., where A > 0. We suppose that X(0) = 1. 
(a) Let T,, := min{t > 0: X(t) =n (> 1)}. That is, T,, is the time needed for 
the number of individuals in the population to be equal to n. Show that the 
probability density function of T,, is given by 
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fr, (t) =A(n— De" — a **)"-? for t > 0 


(b) Let N(t) be the number of descendants of the ancestor of the population 
at time ¢, so that N(t) = X(t) — 1. Suppose that the random variable 7 has 
an exponential distribution with parameter yu. Show that 


PIN(r) =n] = £B(£ +1,n+1) forn =0,1,... 


where B(.,-) is the beta function defined by 


1 
Bay) = f #-1(1 —£)9"! dt 


for x,y € (0, 00). 


Remark. We can show that 


P(x) P(y) 
B(z,y) = = 
(x,y) Tie +y) 
Question no. 81 
A birth and death process, {X (¢), > 0}, has the following birth and death 
rates: 
An =A forn=0,1 and pr=np forn=1,2 


Moreover, the capacity of the system is equal to two individuals. 
(a) Calculate, assuming that \ = py, the average number of individuals in the 


system at a time instant ¢ (large enough), given that the system is not empty 
at this time. 


(b) Calculate the probability that the process will spend more time in state 
0 than in state 1 on two arbitrary visits to these states. 
(c) Suppose that 4; = 0 and that, when X(t) = 2, the next state visited will 


be 0, at rate 24. Write the balance equations of the system, and solve them 
to obtain the limiting probabilities. 


Question no. 82 
Let {X(t),¢ > 0} be a birth and death process whose rates An and [in are 
given by 


An =bLn=nA forn=0,1,2,... 


We set p;,(t) = P[X(t) = k], for all k € {0,1,2,...} and for all t > 0. That 
is, p,(t) denotes the probability that the process will be in state k at time ¢. 
Suppose that p;(0) = 1. It can be shown that 


dM (At)F-2 
t = =_e fi k = 1,2, eee 
pol =T yg and PRO = ype fr 
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(a) Calculate E[X(t) | X(t) > 0]. 
Indication. We have E[X(t)| = 1. 
(b) Calculate the limiting probabilities x; and show that they satisfy the 
balance equations of the process. 
(c) Use the Kolmogorov backward equation satisfied by p1,o(t) to obtain 
p2,0(t). 
Indication. We have pi .9(t) = po(t) above. 
Question no. 83 

We consider a system composed of three components placed in parallel 
and operating independently. The lifetime X; (in months) of component 7 
has an exponential distribution with parameter \, for 7 = 1,2,3. When the 
system breaks down, the three components are replaced in an exponential 
time (in months) with parameter y. Let X(t) be the number of components 
functioning at time t. Then {X(t),¢ > 0} is a continuous-time Markov chain 
whose state space is the set {0,1,2,3}. 
(a) Calculate the average time that the process spends in each state. 
(b) Is the process {X(t),¢ > 0} a birth and death process? Justify. 
(c) Write the Kolmogorov backward equation for po,0(t). 
(d) Calculate the limiting probabilities of the process if \ = p. 


Question no. 84 

Let {N(t),t > 0} be a counting process (see p. 231) such that N(0) = 0. 
When the process is in state 7, the next state visited will be 7 + 1, for all 
j = 0. Moreover, the time 7; that the process spends in state 7 has the 
following probability density function: 


fr;(8) = 2(7 + 1)As e GtDAS* for s > 0 


Finally, the 7;’s are independent random variables. 
Now, consider the stochastic process {X(u),u > 0} defined by 


Oifu <7 


AON = . io ge SUS ay 


fork =1,2,.... 

(a) Show that the stochastic process {X(u),w > 0} is a continuous-time 
Markov chain. 

(b) Calculate P[X(2) = 1]. 

(c) Calculate, if they exist, the limiting probabilities 7; of the stochastic pro- 
cess {X(u),u > O}. 


Question no. 85 
We consider the particular case of the gambler’s ruin problem (see p. 101) 
for which k = 4 and p = 1/2. Suppose that the length T (in minutes) of 
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a play (the outcome of which is the player’s winning or losing $1) has an 
exponential distribution with mean equal to 1/2. Moreover, when the player’s 
fortune reaches $0 or $4, he waits for an exponential time S (in hours) with 
mean equal to 2 before starting to play again, and this time is independent of 
what happened before. Finally, suppose that the player has $1 when he starts 
to play again if he was ruined on the last play of the preceding game and that 
he has $3 if his fortune reached $4 on this last play. 

Let X(t), for t > 0, be the player’s fortune at time t. The stochastic process 
{X(t),t > 0} is a continuous-time Markov chain. 


(a) (i) Is the process {X(t),¢ > 0} a birth and death process? If it is, give its 
birth and death rates. If it’s not, justify. 

(ii) Answer the same question if the player always starts to play again 
with $1, whether his fortune reached $0 or $4 on the last play of the previous 
game. 


(b) (i) Write, for each state 7, the Kolmogorov backward equation satisfied 
by the function po,;(t). 
(ii) Use the preceding result to obtain the value of the sum ya 2,3 (t)- 


(c) Calculate the limiting probabilities of the process {X(t),t > 0}, for all 
states 7. 


Question no. 86 
Let {X,,n = 0,1,...} be a (discrete-time) Markov chain whose state 
space is {0,1,2} and whose one-step transition probability matrix is 


a (1 — a)/2 (1-a)/2 
P= | (1-a)/2 a  (l1-a)/2 
2(1-—a)/3 (l-—a)/3 a 


where a@ & [0,1]. Suppose that the process spends an exponential time with 
parameter A in state i before making a transition, with probability p;;, to 
state j, for 7,7 € {0,1,2}. Let X(t) be the position, that is, the state in which 
the process is, at time t. 

(a) For what value(s) of a is the stochastic process {X(t),t > 0} a continuous- 
time Markov chain? Justify. 


(b) For the value(s) of a in (a), calculate, assuming they exist, the limiting 
probabilities 7;, for j = 0,1, 2. 
Question no. 87 


In the preceding question, suppose that the transition matrix P is instead 
the following: 


a Bl-a-gB 
P= y 0 1-y¥ 
1-68 0 


where a, @, and ¥ € (0,1). 
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(a) For what values of a, 8, and ¥ is the stochastic process {X(t),t > O} a 
birth and death process? Justify and give the value of the parameters vj, j;, 
and j4;, for 7 = 0,1,2. 

(b) For the values of a and @ found in (a), and for y € (0,1), calculate, 
assuming they exist, the limiting probabilities 7;, for all j. 


Question no. 88 

Let {X(t),¢ > 0} be a (continuous-time) stochastic process whose state 
space is the set {0,1}. Suppose that the process spends an exponential time 
with parameter A in a state before making a transition to the other state, 
where A is a discrete random variable taking the values 1 and 2 with proba- 
bility 1/3 and 2/3, respectively. 
(a) Is the stochastic process {X(t),¢ > 0} a continuous-time Markov chain? 
Justify. 


(b) Suppose that X(0) = 0. Let 7) be the time that the process spends in 
state 0 before making a first transition to state 1. Calculate the probability 
P[A=1| 7 < Jj. 


Question no. 89 

Suppose that the continuous-time stochastic process {X(t),¢ > 0}, whose 
state space is {0,1}, spends an exponential time with parameter 1 in a state 
the first time it visits this state. The second time it visits a state, it stays there 
an exponential time with parameter 2. When both states have been visited 
twice each, the process starts anew. 


(a) Is the stochastic process {X(t),t > 0} a birth and death process? Justify. 


(b) Let N be the number of visits to state 0 from the initial time 0, and let 
7 be the time that the process spends in state 0 on an arbitrary visit to this 
state. Calculate approximately P[N is odd | to < 1] if we assume that the 
most recent visit to state 0 started at a very large time f. 


Question no. 90 

A system is composed of three components operating independently. Two 
active components are sufficient for the system to function. Calculate the 
failure rate of the system if the lifetime 7; of component i has an exponential 
distribution with parameter \ = 2, for i = 1, 2,3. 


Question no. 91 
Let {Ni(t),t > 0} and {No(t),¢ > 0} be two independent Yule processes, 
with rates A, = nf; and Ay, = nO, for n =0,1,..., respectively. We define 


X(t) = Ni(t) + No(t) for t >0 


(a) For what values of the constants 0, and @2 is the stochastic process 
{X(t),t > 0} a continuous-time Markov chain? Justify and give the value 
of the parameters v,, of this process. 


(b) For the values of 0, and 2 found in (a), calculate p;,;(t), for 7 > 72> 1. 
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Question no. 92 
We define 
X(t) = |Ni(t) — No(t)| for t > 0 
where {N,(t),t > 0} and {No(t),t > 0} are two independent Poisson pro- 
cesses, with rates Ay = Ao = A. 


(a) Show that {X(t),¢ > 0} is a birth and death process, and give the rates 
An and tn, forn=0,1,.... 


(b) Calculate, if they exist, the limiting probabilities 7;, for 7 = 0,1,.... 
Question no. 93 


Calculate P[X1, < X2 < X3] if X1, X2, and X3 are independent random 
variables such that X; ~ Exp(,,), for i = 1, 2,3. 


Question no. 94 
Let {X(t),t > 0} be a birth and death process whose state space is the 
set {0, 1,2} and for which 


Ao = A, Ay = py = 4/2, and 2 =A 


We consider two independent copies, {X,(t),t > 0} and {X2(t),t > O}, of this 
process, and we define 


Y(t) = |X(t) — Xo(t)| fort >0 


We can show that {Y(t),t > 0} is also a birth and death process. 

(a) Give the birth and death rates of the process {Y(t),t > 0}. 

(b) Calculate the expected value of the random variable Y(t) after two tran- 
sitions if X,(0) = X2(0) = 0. 

(c) Calculate the limiting probabilities of the process {Y(t),t > 0}. 


Question no. 95 
We consider a birth and death process, {X(t),t > 0}, whose state space 
is the set {0,1,2,...} and whose birth and death rates are given by 


An =nA and pp=nu forn=0,1,... 
where A, : > 0. Suppose that X(0) =i € {1,2,...}. Calculate E[X(t)]. 


Indication. We can use the Kolmogorov equations. 


Question no. 96 
The rates A, and py, of the birth and death process {X(t),t > 0}, whose 
state space is the set {0,1,... ,c}, are 


An=(e-—n)A and tn = np 


for n = 0,1,...,c. Suppose that X(0) = k € {0,1,... ,c}. Calculate the 
function p,,-(t). 
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Indication. In the case of a continuous-time Markov chain defined on the set 
{0,1}, and for which 


Ao =A, A; = 0, Lo = 0, and Mj =p 


we have (see p. 131) 


A LL 
t= —(u+A)t 
Poo(t) coder +a 
and : i 
t= = —(u+A)t 
Prolt) B+A TT 


Question no. 97 
Let {X(¢),t > 0} be a birth and death process for which 


1 
al a forn=0,1,... and pf,p=n forn=1,2,... 


and let ft) > 0 be the time instant at which the first birth occurred. 


(a) Suppose that we round off the time by taking the integer part. Calculate 
the probability that the first event from fp will be a birth. 

Indication. If X ~ Exp(A), then 1 + int(X) ~ Geom(1 — e~*), where int 
denotes the integer part. 


(b) What is the probability that there will be at least two births among the 
first three events from to? 


(c) Calculate, if they exist, the limiting probabilities of the process. 


Question no. 98 

The lifetime of a certain machine is a random variable having an expo- 
nential distribution with parameter A. When the machine breaks down, there 
is a probability equal to p (respectively, 1 — p) that the failure is of type I 
(resp., II). In the case of a type I failure, the machine is out of use for an 
exponential time, with mean equal to 1/ time unit(s). To repair a type II 
failure, two independent operations must be performed. Each operation takes 
an exponential time with mean equal to 1/p. 


(a) Define a state space such that the process {X(t),t > 0}, where X(t) 
denotes the state of the system at time ¢, is a continuous-time Markov chain. 


(b) Calculate, assuming the existence of the limiting probabilities, the prob- 
ability that the machine will be functioning at a (large enough) given time 
instant. 


Question no. 99 


A person visits a certain Web site according to a Poisson process with 
rate A per day. The site in question contains a main page and an internal 
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link. The probability that the person visits only the main page is equal to 
3/4 (independently from one visit to another). Moreover, when she clicks on 
the internal link (at most once per visit), the probability that she will return 
(afterward) to the main page is equal to 1/2. We define the states 


0: the person is not visiting the site in question 

1: the person is visiting the main page (coming from outside the site) 
2: the person is visiting the internal link 

3: the person is visiting the main page, after having visited the link 


Let 7 be the time (in hours) spent in state k, for k = 0,1,2,3. We assume 
that the random variables 7, are independent and that 7, has an exponential 
distribution with parameter 1%, for k = 1, 2,3. 

Remark. We suppose that when the process is in state 3, the internal link is 
highlighted and that this highlighting is removed when the person leaves the 
site. 

Let X(t) be the state the process is in at time t > 0. We can show that 
{X(t),t > 0} is a continuous-time Markov chain. 

(a) Give the probabilities p;; of the process. 

(b) Is the process {X(t),t > 0} a birth and death process? If it’s not, is it 
possible to rename the states so that {X(t),t > 0} becomes a birth and death 
process? Justify. 

(c) Calculate the average time spent on the site in question on an arbitrary 
visit, given that 7, = 1/4 for this visit. 

(d) Calculate the limiting probabilities 7;. 


4 


Diffusion Processes 


4.1 The Wiener process 


We already mentioned the Wiener process twice in Chapter 2. In this section, 
we will first present a classic way of obtaining this process from a random 
walk. Then we will give its main properties. 


Consider the discrete-time Markov chain {X,,n = 0,1,...} whose state 
space is the set of all integers Z := {0, +1, +2,...} and whose one-step tran- 
sition probabilities are given by 


pa = PHiai = 1/2. for alla eZ (4.1) 


This Markov chain is a symmetric random walk (see p. 48). A possible interpre- 
tation of this process is the following: suppose that a particle moves randomly 
among all the integers. At each time unit, for example, each minute, a fair coin 
is tossed. If “tails” (respectively, “heads”) appears, then the particle moves 
one integer (that is, one unit of distance) to the right (resp., left). 

To obtain the stochastic process called the Brownian motion, we accelerate 
the random walk. The displacements are made every 6 unit of time, and the 
distance traveled by the particle is equal to € unit of distance to the left or to 
the right, where, by convention, 6 > 0 and € > 0 are real numbers that can be 
chosen as small as we want. As the Wiener process is a continwous-teme and 
continuous-state process, we will take the limit as 6 and € decrease to 0, so that 
the particle will move continuously, but will travel an infinitesimal distance on 
each displacement. However, as will be seen subsequently, we cannot allow the 
constants 6 and ¢€ to decrease to 0 independently from each other; otherwise, 
the variance of the limiting process is equal either to zero or to infinity, so 
that this limiting process would be devoid of interest. 

We denote by X(t) the position of the particle at time t, and we suppose 
that X(0) = 0. That is, the particle is at the origin at the initial time. Let 
N be the number of transitions to the right that the particle has made after 
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its first n displacements. We can then write that the position of the particle 
after nd unit(s) of time is given by 


X(nd) = (2N — n)e (4.2) 


Note that if all the displacements have been to the right, so that N = n, 
then we indeed have that X(néd) = ne. Similarly, if N = 0, then we obtain 
X(nd) = —ne, as it should be. 


Remark. Since the particle only moves at time instants 6, 26, ..., we may 
write that its position at time t is given by 


X(t) = X((t/6)6) for allt >0 (4.3) 


where [ ] denotes the integer part. 


Because the tosses of the coin are independent, the random variable N has 
a binomial distribution with parameters n and p = 1/2. It follows that 


E(X(n6)| == (2x #-n) €=0 (4.4) 
and 


V[X(nd)] =4 2 V[N] =4 2 x =ne (4.5) 


If we first let 6 decrease to 0, then the random walk becomes a continuous- 
time process. However, given that 


VIX (eng = 2 = . x 2 (4.6) 


we find that € must tend to 0 at the same time as 6; otherwise, the variance 
of X(t) will be infinite. Actually, to obtain an interesting limiting process, 6 
and e? must decrease to 0 at the same speed. Consequently, we assume that 
there exists a constant o (> 0) such that 


e=oV5 <> =075 (4.7) 


Thus, when we let 6 decrease to 0, we obtain a process that is also with 
continuous-state space and for which 


E[X(t)] =0 (4.8) 
and 
ViX(t)}) “307% vero (4.9) 


Remark. By choosing € = oV6, we directly have that V[X(t)] = 07, for all 
t > 0. That is, the variance of X(t) is actually equal to o*t for any positive 
value of 6, and not only in the limit as 6 | 0. 
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From the formula (1.113), we may write that 
P[X(n6) < a] ~ P[N(0, ne?) < 2] (4.10) 
from which we deduce that 
P[W(t) < 2] = P[N(0,07t) < 2] (4.11) 
where 


W(t) := lim X(t) (4.12) 


That is, the random variable W(t) has a Gaussian distribution with zero 
mean and variance equal to a? ¢. This result is the essential characteristic of 
the Wiener process. Moreover, since a random walk is a process with inde- 
pendent and stationary increments (see p. 50), we can assert that the process 
{W(t),t > 0} has these two properties as well. 


Based on what precedes, we now formally define the Wiener process. 


Definition 4.1.1. A stochastic process {W(t),t > 0} is called a Wiener 
process, or a Brownian motion, if 


i) W(0) =0, 
it) {W(t),t > 0} has independent and stationary increments, 
iti) W(t) ~ N(0,o7?t) Vt>0. 


Remarks. i) The name Brownian motion is in honor of the Scottish botanist 
Robert Brown.’ He observed through a microscope, in 1827, the purely ran- 
dom movement of grains of pollen suspended in water. This movement is due 
to the fact that the grains of pollen are bombarded by water molecules, which 
was only established in 1905, because the instruments Brown had at his dis- 
posal at the time did not enable him to observe the water molecules. The 
Brownian motion and the Poisson process (see Chapter 5) are the two most 
important processes for the applications. The Wiener process and processes 
derived from it are notably used extensively in financial mathematics. 


ii) Let 


Bit) := Wo). 


(4.13) 
We have that V[B(t)] = t. The stochastic process {B(t),t > 0} is named 
a standard Brownian motion. Moreover, if we sample a standard Brownian 
motion at regular intervals, we can obtain a symmetric random walk. 


1 Robert Brown, 1773-1858, was born in Scotland and died in England. He was a 
member of the Royal Society, in England. 
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iii) The Wiener process has been proposed as a model for the position of the 
particle at time t. Since the distance traveled in a time interval of length 6 
is proportional to Vd, we then deduce that the order of magnitude of the 
velocity of the particle in this interval is given by 


=— —oo whend]0 (4.14) 


5 V5 


However, let V(t) be the average velocity (from the initial time) of the 
particle at time t > 0. It was found experimentally that the model 


for t > 0 (4.15) 


was very good for values of t large enough (with respect to 4). We will see 
another model, in Subsection 4.2.5, which will be appropriate for the velocity 
of the particle even when ¢ is small. 


iv) We can replace condition iii) in Definition 4.1.1 by 
W(t +s) —-W(s)~N(0,o7t) Vs,t>0 (4.16) 


and then it is no longer necessary to assume explicitly that the process 
{W(t),t > 0} has stationary increments, since it now follows from the new 
condition iii). 
v) Let 

W*(t):=W(t)+e (4.17) 


where c is a real constant, which is actually the value of W*(0). The process 
{W*(t),t > 0} is called a Brownian motion starting from c. We have that 
W*(t) ~ N(c,o7t), Vt > 0. We could also consider the case when c is a 
random variable C independent of W(t), for all t > 0. Then we would have 


E[W*(t)] = E[C] and V[W*(t)] =07t + VIC] (4.18) 


vi) Wiener proved the following very important result: W(t) is a continuous 
function of ¢ (with probability 1). Figure 4.1 shows a (simplified) example of 
the displacement of a particle that would follow a Brownian motion. In reality, 
the trajectory of the particle would be much more complicated, because there 
should be an infinite number of changes of direction of the particle in any 
interval of finite length. We can thus state that the function W(t) is nowhere 
differentiable (see, however, Section 4.3). 


In general, it is very difficult to explicitly calculate the kth-order density 
function (see p. 49) of a stochastic process. However, in the case of the Wiener 
process, {W(t),t > 0}, we only have to use the fact that this process has 
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Wit) 


Fig. 4.1. (Simplified) example of the Brownian motion of a particle. 


independent and stationary increments. We can thus write, for t) < to < 
... <tr, that 


é 
f (lige se WE yep te) f(t) [] fe; —wy-13tj —tjy-1) (4.19) 


j=2 
where 
Cn) eee a | oe alt wie (4.20) 
ame ror aa Ec 


is the density function of a random variable having a Gaussian N(0,07t) dis- 
tribution. Indeed, we have 


—» 


{W(t;) = w;} (4.21) 


J 
k 
= {W(t) = wih) ¢ 0) {W(ts) — W(ty-1) = wy — wy-1} 
j=2 


and the random variables W(t,), W(t2) - W(t), ... are independent and 

all have Gaussian distributions with zero means and variances given by ot, 
2 

o (tg — 1), sees 


Remark. We deduce from what precedes that the Wiener process is a Gaussian 
process (see Section 2.4). It is also a Markovian process, because it is the limit 
of a Markov chain. 


To calculate the autocovariance function (see p. 49) of the Wiener process, 
note first that 
Cov[X,Y + Z] = E[X(Y + Z)] — E[X]E[Y + Z] 
= {E[XY] — E[X]E[Y]} + {E[X2Z] — E[X]E[Z]} 
= Cov[X, Y] + Cov[X, Z] (4.22) 
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Then we may write that 


Cw(t,t+s) = Cov[W(t),W(t+s)] 
= Cov(W(t),W(t)+W(t+s)—- W(t)] 
= Cov(W(t),W(t)] + Cov[W(t), W(t +s) — W(t)] 
ind. inet’ Cov iW(t), W(t)] = V[W(t)] = 07¢ (4.23) 


for all s,t > 0. This formula is equivalent to 
Cw(s,t) =o? min{s,t} for all s,t >0 (4.24) 


Remarks. i) The random variables W(t) and W(t + s) are not independent, 
because the intervals {0,t] and [0,t+ s] are not disjoint. Thus, the larger W(t) 
is, the larger we expect W(t + s) to be. More precisely, we have 


W(t+s)| W(t) ~N(W(t),o7s) Vs,t>0 (4.25) 


ii) Since the function Cy/(t,t + s) is not a function of s alone, the Wiener 
process is not stationary, not even in the wide sense (see p. 52). As we already 
mentioned, the notion of processes with stationary increments and that of 
stationary processes must not be confounded. 


iii) Since E[W(t)] = 0, we also have 
Rw(s,t) =o? min{s,t} for all s,t >0 (4.26) 


where Ry(-,-) is the autocorrelation function of the process {W(t),t > O} 
(see p. 49). 


Example 4.1.1. If the random variables W(t) and W(t+ s) were independent, 
we would have W(t) + W(t+s) ~ N(0,07(2t+ s)), which is false. Indeed, we 
may write that 


W(t)+W(t+s)=2W(t)+ [W(t+s)-W(j)]=X+Y 


where X := 2W(t) and Y := W(t +s) — W(t) are independent random 
variables, because the Wiener process has independent increments. Moreover, 
we have 

E|X] = E[Y] =0 


and 
V[X]) =4V[W(t)]} =407t and V[Y] =V[W(s)] =o7s 


(using the fact that the Wiener process also has stationary increments). We 
thus have 
W(t) +W(t+s) ~N(0,07(4t +4 s)) 


which follows directly from the formula (1.108) as well. 
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Remark. As the increments of the Wiener process are stationary, the random 
variables W(t + s) — W(t) and W(s) (= W(s) — W(0)) have the same dis- 
tribution. However, this does not mean that W(t + s) — W(t) and W(s) are 
identical variables. Indeed, suppose that t = 1, s = 1, and W(1) = 0. We 
cannot assert that W(2) —-W(1) = W(2) —0 =0, since P[W(2) = 0] = 0, by 
continuity. 


Given that the Brownian motion is Gaussian and that a Gaussian process is 
completely determined by its mean and its autocovariance function (see p. 59), 
we can give a second way of defining a Brownian motion: a continuous-time 
and continuous-state stochastic process, {X(t),t > 0}, is a Brownian motion 


i) X(0) = 0, 
ii) {X(t),t > 0} is a Gaussian process, 
iii) E[X(t)] = 0, 


iv) Cx(s,t) = Cov[X(s), X(t)] = o? min{s,t} V s,t > 0, where o > Oisa 
constant. 

It is generally easier to check whether the process {X(t),t > 0} possesses 
the four properties above, rather than trying to show that its increments are 
or are not independent and stationary. This second definition of the Brownian 
motion is particularly useful when {X(t),t > 0} is some transformation of a 
Wiener process {W(t),t > 0}. As we saw in Section 2.4, any affine transfor- 
mation of a Gaussian process is also a Gaussian process. That is, if 


X(t) =e W(t) +¢0 (4.27) 


where co and c; # 0 are constants, then {X(t),t > 0} is a Gaussian process. 
Moreover, if we only transform the variable t, for example, if 


X(t) = W(t?) (4.28) 


then the process {X(t),t > 0} is also a Gaussian process. 


Remark. We could drop the first condition above if we accept that a Brownian 
motion can start from any point wo € R. Similarly, we could replace the third 
condition by E[X(t)} = ut, where pz is a real constant. In this case, the 
stochastic process {X(t),¢ > 0} would be a Brownian motion with drift ys (see 
Subsection 4.2.1). 


Example 4.1.2. Let {W(t),t > 0} be a Brownian motion. We set 
X(0)=0 and X(t)=tW(1/t) ift>0 


At first sight, the stochastic process {X(t),t > 0} does not seem to be a 
Wiener process. However, we have 
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E[X(t)] =tE(W(1/t)]) =t-0=0 ift>0 
because E[W(t)] = 0, for any value of t > 0, and then 


Cx(s,t) = E[X(s)X(t)] -0 x0 
= E[s W(1/s) t W(1/t)] = st Cw(1/s, 1/t) 
= sto? min{1/s,1/t} = 0? min{s, t} 
Moreover, we can assert that {X(t),t > 0} is a Gaussian process, because here 
X(t) is a linear transformation of W(1/t). Since X(0) = 0 (by assumption), 


we can conclude that {X(t),t > 0} is a Brownian motion having the same 
characteristics as {W(t), t > O}. 


Remark. We must not forget that the variable t is deterministic, and not 
random. Thus, we can consider it as a constant in the calculation of the 
moments of the process {X(t),t > 0}. 


Example 4.1.3. We define the stochastic process {X(t),t > 0} by 
X(t) = B(t)|{B(t) >0} fort >0 


where {B(t),t > 0} is a standard Brownian motion. 
(a) Show that the probability density function of X(t) is given by 


fx(y(Z) =2fayy(z) forz 20 


(b) Calculate (i) E[X(t)] and (ii) V[X(d)], for t > 0. 

(c) Is the stochastic process {X(t),t > 0} (i) Gaussian? (ii) wide-sense sta- 
tionary? Justify. 

(d) Are the random variables X(t) and Y(t) := |B(t)| identically distributed? 


Solution. (a) We may write that 


faqy(2) 


fx) (2) = PIB(t) > 0] = 2 fay) (2) for x >0 


because B(t) ~ N(0,t) > P[B(t) > 0] = 1/2. 
(b) (i) We calculate 


/2 poo 

@i fo 2 2t\" we ssa 
E(Xx(t)] = Bae “Maz = - (2) BAY 22/4 ay 
x)? [ace z : ae) 


Loe) 


=- (2t/)\/? ene tt} = (2t/)'/? fort >0 
0 
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(ii) Notice first that 
E|B?(t)] = E[B?(t) | B(t) > 0|P[B(t) > 0] + E[B?(t) | B(t) < 0] P[B(t) < 0] 
FBO sa) 
1/2 
Since E[B?(t)] = V[B(t)] =t and E[B?(t) | B(t) > 0] = E[B?(t) | B(t) <0] 
(by symmetry and continuity), it follows that 
vix(t)] 2 B[B2(t) | BO) )>0)-2=1-2 for t > 0 


(c) (i) Since X(t) > 0 V t > 0, the stochastic process {X(t),t > O} is not 
Gaussian. 

(ii) It is not WSS either, because E|X(t)] is not a constant. 
(d) We have 


PIV (t) < y] "=" P[-y < Bit) < 9] = Faw (y) — Faw (-y) 
sym. 


= fryw ly) =2fawly) fory20 


Thus, X(t) and Y(t) are identically distributed random variables. 


4,2 Diffusion processes 


Continuous-time and continuous-state Markovian processes are, under certain 
conditions, diffusion processes. The Wiener process is the archetype of this 
type of process. One way, which can be made even more rigorous, of defining 
a diffusion process is as follows. 


Definition 4.2.1. The continuous-time and continuous-state Markovian sto- 
chastic process {X(t),t > 0}, whose state space is an interval (a,b), is a 
diffusion process if 


linn = PIIX(t +6) -X(t)| > | X(t) =a] =0 (4.29) 


Vd>0 andV z € (a,b), and if its infinitesimal parameters defined by (see 
p. 63) 


m(a;t) = lim ~E|X(t +e) — X(t) | X(t) = 2] (4.30) 
and 
v0 t) = lim ~E(X(t +6) — X(t)? | X() =a] (4.31) 


are continuous functions of x and of t. 
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Remarks. i) The condition (4.29) means that the probability that the process 
will travel a distance greater than a fixed constant 6 during a sufficiently short 
period of time is very small. In practice, this condition implies that X(t) is a 
continuous function of t. 

ii) We assumed in the definition that the infinitesimal mean m(z;t) and the 
infinitesimal variance v(z;t) of the process exist. 

iii) The state space Sx (4) of the stochastic process {X(t),t > 0} may actually 
be any interval: [a,b], (a,], [a,6), or (a,b). Moreover, if the interval does 
not contain the endpoint a, then a may be equal to —oo. Similarly, we may 
have that 6 = oo if Sy) = (a,b) or [a,6). Finally, if Sx() = [a@, 6], then the 
functions m(z;t) and v(x;t) must exist (and be continuous) for a < x < b 
only, etc. 


Example 4.2.1. In the case of the Wiener process, {W(t),t > 0}, we have that 
W(t+e) | {W(t) =w}~ N(w,o7e), for all ¢ > 0. We calculate 


iperrer- W(H)| > 8 | W(t) =u] = lim =PIIN(,076)| > 5 


e10 
“ip [sean> i] mn? (1-0(4) 
where 
@(x) := / . seer dz (4.32) 


is the distribution function of the N(0,1) distribution. We may write that 


(x) = 5 (: +erf (=)) (4.33) 


where erf(-) is the error function. 
Finally, making use of the formula (4.33) and of the asymptotic expansion 


2 
e* {1 1 
erf(z) me iE-S +} 


which is valid for z > 1, we find that 
lim? {1-0(—.) | =0 
elO € ove 


. 1 All 
lim =B[W(t + €) - W(t) | W(t) = w] = lim=0.=0 


Next, we have 
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and 
lim “EW t+e)-—W(t))? | W(t) =u] 
= lim “52, where Z ~ N(0,07e) 


1 1 
= lim -V[Z] = lim = oe = 0? 
ej0 € el0 € 
Thus, we have m(x;t) = 0 and v(z;t) = 0. Because the infinitesimal param- 


eters of the Wiener process are constants, the functions m(z;t) and v(z;t) are 
indeed continuous. 


The most important case for the applications is the one when the diffusion 
process { X(t), t > 0} is time-homogeneous, so that the infinitesimal moments 
of {X(¢),t > 0} are such that m(z;t) = m(z) and v(z;t) = v(x). We can then 
assert (see p. 64) that the process {Y(t),¢ > 0} defined by 


Y(t) = g[X(@)] fort >0 (4.34) 


where g is a strictly increasing or decreasing function on the interval [a, 6] = 
Sx ¢) and such that the second derivative g’’(z) exists and is continuous, for 
all x € (a,b), is also a diffusion process, whose infinitesimal parameters are 
given by 


my(y) = m(a)g(e) + Solx)g"(x) and vy(y)=v(a)lo'(e)P (4.35) 


where the variable x is expressed in terms of y: x = g™(y) (the inverse 
function of g(z)). Moreover, we have that Sy(z) = (g(a), 9(b)] if g is strictly 
increasing, while Sy(z) = [g(6), g(a) if g is strictly decreasing. 


Remarks. i) The function g must not be a function of the variable t. 


ii) We assume in what precedes that the process {X(t),t > 0} can move from 
any state x € (a,b) to any other state y € (a,b) with a positive probability. 
We say that {X(t),t > 0} is a regular diffusion process. Then the process 
{Y(t),t > 0} is regular as well. 


4.2.1 Brownian motion with drift 


A first important transformation of the Wiener process is a generalization of 
this process. Let 


Y(t) := 0 B(t) + pt (4.36) 


where {B(t),t > 0} is a standard Brownian motion, and p and o ¥ 0 are real 
constants. Note that in this case the function g would be given by 
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g(xz,t) = ox + pt (4.37) 


Thus, we cannot use the formulas (4.35) to calculate the infinitesimal param- 
eters of the process {Y (t),t > 0}. However, we have 


EY (t) | Y(to) = yo] = yo + w(t — to) (4.38) 
and 
VY (t) | ¥(to) = yo] = 07(t - to) (4.39) 


for all t > to. We then deduce from the formulas (2.58) and (2.59) that 


my(y)=p and vy(y) =o? for ally (4.40) 


Remark. If we try to calculate the function my(y) from (4.35), treating ¢ as 
a constant, we find that my(y) = 0, which is false, as we see in Eq. (4.40). 


Definition 4.2.2. Let {Y(t),t > 0} be a sani pes whose infinitesimal 
parameters are given by my (y) = uw and vy(y) = 07. The process {Y (t),t = 0} 
is called a Brownian motion (or Wiener process) with drift p. 


Remarks. i) The parameter p is the drift coefficient, and o? is the diffusion 
coefficient of the process. The term parameter, rather than coefficient, is used 
as well. 


ii) If the random walk had not been symmetric in the preceding section, we 
would have obtained, under some conditions, a Wiener process with nonzero 
drift coefficient p. 


iii) Since the function f{B(t)] := oB(t) +t is an affine transformation of the 
variable B(t) ~ N(0,t), we may write 


Y(t) ~ N(ut, 07t) (4.41) 
or, More generally, 
Y(t) | {¥ (to) = yo} ~ N(yo + w(t —to),07(t—to)) Vt> to (4.42) 


Moreover, the process {Y(t),t > 0} is a Gaussian process having independent 
and stationary increments. It follows (with Y(0) = 0) that 


EY(t+s)¥()] = Bl(¥(t+s)- YW) +YO)¥O) 
ind._iner. Ely(t +s) —Y (HEY (t)] + E{Y?(t)] 
stat.incr. ElY (s)|E[Y (t) exer )| 
i (us)(ut y+ (o 24 a jee) (4.43) 


which implies that 
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Cy (t+ s,t) = Cov[Y(t+s), ¥(t)] = E[Y(t+ s)Y(t)] — E[Y (t+ s)|E[Y (4) 
= p'sttort+ pu - ~p(t+s) pt = 0°t Vs,t>0 (4.44) 


Thus, the Brownian motion with drift has the same autocovariance function 
as the Wiener process. 


iv) The conditional transition density function p(y, yo;t, to) (see p. 62) of the 
Brownian motion with drift coefficient jz and diffusion coefficient o? satisfies 
the partial differential equation (see p. 64) 


= +p —-— 5 =0 (4.45) 


as well as the equation 


Op Op 0 Op 
Op OP 5 4.46 
Bip + Ban Aya ae. aye (4.46) 


We can check that 


- 1 [y — (yo + u(t — to))]? 
P(Y, Yost, to) = Janotgaty 173 pope (4.47) 


for y, yo € R and t > tp > 0. We have 


lim p(y, yo; t, to) = 5(y — yo) (4.48) 


which is the appropriate initial condition. 


4.2.2 Geometric Brownian motion 


A diffusion process that is very important in financial mathematics is obtained 
by taking the exponential of a Brownian motion with drift. 

Let {X(t),t > 0} be a Wiener process with drift coefficient yz: and diffusion 
coefficient a7. We set 


Y(t) =e*) fort >0 (4.49) 


Since the function g(x) = e” does not depend on t, we deduce from (4.35) 
that 


1. 1 
my (y) = pe* + x7 = pyt 50 8 (4.50) 
and 


vy (y) = 02(e*)? = 0?y? (4.51) 
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Definition 4.2.3. The stochastic process {Y(t),t > 0} whose infinitesimal 
parameters are given by my(y) = («+ 407)y and vy(y) = oy? is called a 
geometric Brownian motion. 


Remarks. i) The state space of the geometric Brownian motion is the interval 
(0,00), which follows directly from the definition Y(t) = eX). The origin is 
a natural boundary (see Section 4.4) for this process. It is used in financial 
mathematics as a model for the price of certain stocks. 

ii) Since Y(t) > 0, for all t > 0, the geometric Brownian motion is not a 
Gaussian process. For a fixed t, the variable Y(t) has a lognormal distribution 
with parameters pt and o7t. That is, 


fyayy) = mae for y > 0 (4.52) 


1 
ae et 
V2r02ty { 207t 


iii) We can generalize the definition of the geometric Brownian motion {Y(t) 
t > 0} by setting 


Y(t) =Y(0)e* (4.53) 
where Y(0) is a positive constant. As in the case of the Wiener process, the 


initial value Y (0) could actually be a random variable. 


iv) To obtain the conditional transition density function p(y, yo; t,to) of the 
process, we can solve the Kolmogorov forward equation 


Op 1 ,\ a a @ 
at = 4.54 
+ (45 50°) p) O dy? a5 (y? p)=0 ( ) 


In the particular case when tg = 0, we find that the solution of this partial 
differential equation that satisfies the initial condition 


lim p(y, yo; t) = 3(y — yo) (4.55) 
is 
1 (In # — pt)? 
piece ee eae Teas 4.56 
Ply, Yo; t) not y exp 202t ( ) 


for y, yo > 0 andt > 0. 


v) The geometric Brownian motion is appropriate to model the evolution of 
the value of certain stocks in nee mathematics when we assume that the 
ratios 


X, XX Xz 


megs Oy Tye 4.57 
Set a (4.57) 
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where Xo is the initial price of the stock and X, is the price after k unit(s) 
of time, are independent and identically distributed random variables. 


We have used the expression diffusion process at the beginning of this 
subsection. By definition, this means that the geometric Brownian motion 
must be a Markovian process, which we now show. 


Proposition 4.2.1. The geometric Brownian motion is a Markovian process, 
whose conditional transition density function is time-homogeneous. 


Proof. We have, in the general case when Y(t) := Y(0)e*™, 


Y(t + s) a Y (0)e* (t+) = Y (0)eX ¢+8)-X+X() 
— Y(t+s)=Y¥(t)e*@+9)-*O (4.58) 
We then deduce from the fact that the Wiener process has independent in- 
crements that Y(t +s), given Y(t), does not depend on the past. Thus, the 
process {Y(t),t > 0} is Markovian. 
We also deduce from the equation above and from the independent and 
stationary increments of the Wiener process that 


PLY (to +8) <y| Y(to) = yo] = P[Y (to)e*t9-*) < y | ¥(to) = yo! 
= Ple* (to+s)-X (to) < y/yo| 
= PleX'*) < y/yo] (4.59) 


for all yo,y > 0 and for all tp,s > 0, from which we can assert that the 
function p(y, yo; t, to) of the geometric Brownian motion is such that 


p(y, yo; t, to) = p(y, yoit—to) O (4.60) 
Remarks. i) Equation (4.58) implies that 
Y(t+s) £Y(t)e*) (4.61) 


That is, the random variable Y(t +s) has the same distribution as Y (t)e*?. 


ii) To prove that the conditional transition density function of the geometric 
Brownian motion is time-homogeneous, we can also check that the function 
[see (4.56)] 


In % — p(t — to)]? 
p(y, yo; t, to) = Cea} (4.62) 


fas ee ee 
2no2(t —to)y r 207(t — to) 


for y, yo > O and t > tg > 0, satisfies the Kolmogorov forward equation (4.54), 
subject to 
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lim p(y, yo; t, to) = 5(y — yo) (4.63) 


From the formula Y(t) = e*, and making use of the formula for the 
moment-generating function (see p. 19) of a random variable X having a 
Gaussian N(j:, 0) distribution, namely 


Mx(s) = E{e?*] = exp {su + 380° (4.64) 


we find that the mean and the variance of Y(t) are given by 


E[Y (t)] = exp { (u + 50°) i} (4.65) 
and 
VIY (t)] = exp { (u + 30°) 2} et = 1) (4.66) 


for all t > 0. Note that E[Y(0)] = 1, which is correct, since Y(0) = 1 (because 
X (0) = 0, by assumption). More generally, for all t > 7, we have 


E Fal = exp { (u + 57°) (t - »} (4.67) 


V Bal = exp { (u + 57°) 2(t — nh (ert) - 1) (4.68) 


We also find, using the fact that the process is Markovian, that 


and 


BYU) [Y(s),0 <8 <1] = BY 1Y(7)] = Yew { (w+ 502) ¢- 7} 
(4.69) 


and 


El¥?(t) | ¥(s),0 <8 <7] = E[¥7(t) | ¥(r)] = ¥7(r) exp{2u + ot a) 
10 


for all t > 7, from which we can calculate V[Y(t) | Y(s),0 < s <7]. 
Finally, we can write (see Ex. 4.1.1) that if Y(0) = 1, then 


EIY(t+s)Y()] = £ lexerenO) ap [ener exe 
= Mx(1), where X ~ N(u(2t + 8), a?(4t + s)) 
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1 5 
= exp 4 j(2t + 8) + 57 (4t + s) (4.71) 


from which, V s,t > 0 we have 
Cy(t+s,t) = EV (t +s)¥()] — EV (t+ IEIY (0) 


1 
= exp {u(at +s)+ 50 (At +s) 


= exp {m(at +3)+ 50 (at + »} — exp { (1 + 57°) (2t + »} 


= exp { (u + 57°) (2t + »} (ev = 1) (4.72) 


Remarks. i) Contrary to the Wiener process, the geometric Brownian motion 
is not a process with independent and stationary increments. Indeed, 


Y(t+s)—Y¥(s) =Y(0)e*“+*) — V(o)e*) = ¥(0)(eX +) — eX) (4.73) 
does not have the same distribution as 
Y(t) —Y(0) =Y(0)e* — Y(0) = Y(0)(e*™ — 1) (4.74) 
for all s > 0. Moreover, the random variables 
Y(t+s)—Y(t) =YV(0)eX"+®) — y(0)eX = Y(0)(eX 49) — eX) (4.75) 


and Y(t)—Y(0) are not independent. To justify this assertion, we can calculate 
the covariance of these variables. Since Y(0) is not random, we have 


Cov[Y (t + s) — Y(t), Y(t) — Y(0)] 
= {E[Y (t+ s)¥(t)] - YO) E(t + s)] ~ EIV2()] + YEW (H]} 
— {EY (t + s)] -— BLY (t)]} {E[¥ ()] — ¥(0)} 
= Cov[Y (t+ s), Y(t)] — VIY(®)] (4.76) 
Using the formulas (4.66) and (4.72), we may write that 


Cov[Y (t +s) — Y(t), Y(t) — ¥(0)] 


no") ol pede) eso eevee) 
= (e*** = 1) exp { (u n 3”) ar} lex { (u + 3°”) s| - | (4.77) 


which is different from zero if s,t > 0, so that the random variables considered 
are not independent. 
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ii) If the drift coefficient jz of the Wiener process is equal to —307, then the 
corresponding geometric Brownian motion is such that [see the formula (4.69)] 


EIY(t) | ¥(s),0<s<7]=Y(r) (4.78) 


Thus, the process {Y(t),t > 0} is a martingale (see p. 102). We can always 
obtain a martingale from an arbitrary Wiener process, by setting 


Y*(t) = Y(0) exp {x - (u + 52°) tS = exp - (u + 30°) r} Y(t) 
(4.79) 


where Y(t) is defined in (4.53). Indeed, we then have 


= Y(r)exp - ( + 50°) rh =Y*(r) (4.80) 


iii) There exists a discrete version of the geometric Brownian motion used, in 
particular, in financial mathematics. Let Y, := InX,, where X,, is the price 
of the shares of a certain company at time n € {0,1,...}. We assume that 


Yn = b+ Yn-1 + €n (4.81) 


where 4 is a constant and the €,’s are independent random variables. Then 
we have 


n 
Yra=npt+ > e+Yo (4.82) 
i=1 


If we now assume that €, ~ N(0,07), for all n, and that Yo is a constant, we 
obtain 


E[YnJ=nput+Yo and V[Y¥,]=no? (4.83) 


The discrete-time and continuous-state processes {Y,,n = 0,1,.. .} and 
{X,,n = 0,1,...} are called a discrete arithmetic Brownian motion and a 
discrete geometric Brownian motion, respectively. 
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4.2.3 Integrated Brownian motion 


Definition 4.2.4. Let {Y(t),t > 0} be a Brownian motion with drift coeffi- 
cient 1 and diffusion coefficient 07, and let 


Z(t) = Z(0) + . Y(s) ds (4.84) 


The stochastic process {Z(t),t > 0} és called an integrated Brownian mo- 
tion. 


Proposition 4.2.2. The integrated Brownian motion is a Gaussian process. 


Proof. First, we use the definition of an integral as the limit of a sum: 
t< tk 
Z(t) = Z(0)+ lim = SCY (=) (4.85) 
no nN a n 


Since the Wiener process is a Gaussian process, the Y(tk/n)’s are Gaussian 
random variables. From this, it can be shown that the variable Z(t) has 
a Gaussian distribution and also that the random vector (Z(t1), Z(t2),... 5 
Z(tn)) has a multinormal distribution, for all t),t2,...,t, and for any n, so 
that the process {Z(t),t > 0} is Gaussian. O 


Remark. We can write that 


Z(tk) = (2 (te) — Z(te-1)) + (4 (te-1) — 2(tk-2)) + 


+ (Z(te) — Z(t1)) + Z(t1) (4.86) 
for k = 2,...,n, where we may assume that t) < tg <--- < tn. We have 
tk 
Z(tk) — Z(th-1) = f) Y(s) ds (4.87) 
tk—-1 


However, even though the increments of the Wiener process are independent, 
the random variables (Z(t2) — Z(t,)) and Z(t;), etc., are not independent, as 
will be seen further on. Consequently, we cannot proceed in this way to prove 
that the integrated Brownian motion is Gaussian. 


Assuming that Z(0) is a constant, we calculate 


ot 


E[Z(t)] = Z(0) + i E|Y (s)\ds = Z(0) + [ (¥(0) +s) ds 


= Z(0)+Y¥(0)t+ Ds (4.88) 


For the sake of simplicity, suppose now that {Y(t),t > 0} is a standard 
Brownian motion (starting from 0) and that Z(0) = 0. Making use of the 


192 4 Diffusion Processes 


formula (see p. 178) E[Y(s)Y(t)] = min{s,t} (because E[Y(t)] = 0 and o = 
1), we may write that 


E[Z(t +s)Z(t)| = E If ec au [PY ¥(v) de] 
=2/f [vere (v) dudo| 
= [ ever) dudv 
= | { | Tinta ara |  min(aso} du} dy 
= [ff vas [va du 
= [ {F+eets-u} dv 


= ¢ € + 3) (4.89) 


From the previous formula, we obtain, under the same assumptions as 
above, that 


Cov[Z(t +s) — Z(t), Z(t)] = E {[Z(t + s) — Z(t)|Z(t)} - 
= E[Z(t+s)Z ae E|zZ? ie 
t 8s 3 
Bien ue (4.90) 


which implies that Z(t + s) — Z(t) and Z(t) are not independent random 
variables V s > 0, and 


E [{Z(t + s) — Z(s)}?] = E[Z°(t + s)] + E[Z?(s)] — 2E[Z(t + s)Z(s)] 


(t+s)> 8? ofs.,t Pe ay 
= —e ~ —_ =e t 
3 + 3 2s 3 o 5 3 +t*s 
3 
#5 = BIZ) (4.91) 


from which we deduce that Z(t +s) — Z(s) and Z(t) are not identically dis- 
tributed random variables. 


We can generalize these results and state the following proposition. 


Proposition 4.2.3. The increments of the integrated Brownian motion are 
neither independent nor stationary. 
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Next, by definition of Z(t), we may write that 


t+s t t+s 
Z(t+s) = Z(0) +f Y(r) dr = Z(0) +f Y(t) dr + Y(r) dr 
0 0 


= Z(t) + / - Y(r) dr (4.92) 


Proposition 4.2.4. The integrated Brownian motion is not a Markovian pro- 
cess. However, the two-dimensional stochastic process (Y(t), Z(t)) is Marko- 
vlan. 


Proof. The value of the last integral above does not depend on the past if 
Y(t) is known, because the Wiener process is Markovian. Now, if we only 
know the value of Z(t), we do not know Y(t). Thus, for the future to depend 
only on the present, we must consider the two processes at the same time. 0 


When {Y(#),¢ > 0} is a standard Brownian motion, the joint probability 
density function of the random vector (Y(t), Z(t)), starting from (yo, 20), is 
given by 


P(Y, Z, Yo; 203t) 
PIV (t) € (yy + dy], Z(t) € (z,2 +d] | YO) = yo, (0) = 20] 
dydz 


2 6 6 
“ay — yo)? + py yo)(z — 20 — yot) pl? 2 — int)?} 
(4.93) 


for y,z,Yo, 20 € R andt > 0. 
Remarks. i) The density function p(y, z, yo, 20;t) is a particular case of the 
joint conditional transition density function p(y, Z, Yo, Z0;t, to). 
ii) To be more rigorous, we should take the limit as dy and dz decrease to 0 
above. However, this notation is often used in research papers. 

The function p(y, Z, yo, 20;t) is the solution of the partial differential equa- 
tion (namely, the Kolmogorov forward equation) 


2 


re) 6) 1 
ait aaparceee 4.94 
a YazP 9 Oy? B= coe) 


which satisfies the initial condition 


+4+—0)— _ f 0 ify A yo or z F 2 
P(Y, 2, Yo, Zoit = 0) = 5(y — Yo, 2 — 20) = ie if y = yo and z = % 490) 
Indeed, taking the limit as t decreases to zero in the formula (4.93), we obtain 


lim p(y, 2, Yo, 20;t) = d(y — yo, 2 — 20) (4.96) 
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Remark. Actually, it is preferable to define the two-dimensional Dirac delta 
function by 


oly — Yo, 2% — zo) =0 if (y, z) # (yo, Zo) (4.97) 


and 


[oo lo) 
i / 5(y — yo, z — 2) dy dz=1 (4.98) 
—-Mwdv-C 
We can then write that 


d(y — yo, 2 — 20) = 5(y — yo)d(z — 20) (4.99) 


Note that we also obtain the formula above, even if we set 5(0,0) = 6(0) = 00, 
by assuming that 


5(0)5(z — 20) = 6(y — yo)6(0) =0 if z Az and y F yo (4.100) 


The formula for the function p(y, z, yo, 20;t) is easily obtained by using the 
fact that it can be shown that the random vector (Y(t), Z(t)) has a bivariate 
normal] distribution. Moreover, the bivariate normal distribution is completely 
characterized by the means, the variances, and the covariance of the two 
variables that constitute the random vector. Here, when {Y(t),t > O} is a 
standard Brownian motion starting from Y(0) = yo, and Z(0) = zo, we deduce 
from what precedes that 


E(Y(t)}=yo, V[Y(t)])=t, and E[Z(t)] = zo + yot (4.101) 


Furthermore, since E[Y(s)Y(t)] = min{s,t} + yg when Y(0) = yo, we find 
that the generalization of the formula (4.89) is 


t 
E[Z(t + s)Z(t)] = t? (5 + 7 + yet? + 22 + 2zoyot (4.102) 


so that 


V(Z(t)] = E[Z?(t)] — {E[Z(e)]P? 
3 3 
= ; + yet? + 22 + 2zoyot — (20 + yot)? = . (4.103) 


Remark. The variance of Z(t) is thus independent of the initial values 2(0) 


and Y(0), as we could have guessed. 


Next, we calculate 


EY (t)Z(t)] = E re { ze i Y(r) ar} 
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t 
= w20+ f BIV()Y()] ar 
0 


t . t? 
= yoZo +f (7 + y2) dr = yozo + 3 + yPt (4.104) 
0 


which implies that 


Cov[Y (t), Z(¢)] = E[Y (t)Z(t)] — E[Y (t)]E[Z(8)] 
2 2 
= ° + yoz0 + yet — yo(20 + yot) = 5 (4.105) 
and 
Vv 2 
PY (t),Z(t) “= peouilduleaul ne ne = v3 (4.106) 


VIY@VIZ@)] P/v3 2 


The expression for the function p(y, z, yo, Z9;t) is obtained by substituting 
these quantities into the formula (1.82). 


We can check that if we replace t by t — to in the formula (4.93), then the 
function p(y, Z, Yo, Z0;t, £9) thus obtained is also a solution of Eq. (4.94), such 
that 


lim p(y, 2, Yo: 20;t, to) = 5(y — yo, 2 — 20) (4.107) 


Consequently, we can state the following proposition. 


Proposition 4.2.5. The integrated Brownian motion is time-homogeneous. 
That is, 


PCY; 2, Yo, 203t, to) = p(y, z, Yo, z0i3t —to) Vt>to >0 (4.108) 


Finally, in some applications, we consider as a model the Brownian mo- 
tion integrated more than once. For example, the doubly integrated Brownian 


motion 
(0+ f Z(s)as (0) + 2(0) ee f [vem \dr ds (4.109) 


where {Y (¢),¢ > 0} is a Wiener process with drift coefficient 4 and diffusion 
coefficient o?, is a Gaussian process for which 


E[D(t)] = D(0) + Z(0)t + [ i + ut)dr ds 
= D(0) + Z(0 t+ V5 ear (4.110) 


6 
We also find that 
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V[D(t)] = o*— (4.111) 
Moreover, the three-dimensional process (Y(t), Z(t), D(t)) has a multinor- 


mal (namely, trinormal or three-variate normal) distribution whose vector of 
means m and covariance matrix K (see p. 59) are given by 


Y(0)+pt ; 
m = Z(0) + Y(O)t+ p> (4.112) 
3 
D(0) +2(0)t + ¥(0)5 + ye 
and 
1 +t/2 t?7/6 
K =o7t | t/2 t?/3 t3/8 (4.113) 
t?/6 t?/8 4/20 


Example 4.2.2. In Example 4.1.1, we calculated the distribution of the sum 
W(t) + W(£+s) by using the fact that the increments of the Wiener process 
are independent. Since the integrated Brownian motion does not have this 
property, we cannot proceed in the same way to obtain the distribution of Z := 
Z(t) + Z(t +s). However, we can assert that Z has a Gaussian distribution, 
with mean (if 4 = 0 and Y(0) = Z(0) = 0) 


E[Z] =0+0=0 


and with variance (if 0? = 1) 


V[Z] = V[Z(t)] + V[Z(t + s)] + 2 Cov[Z(t), Z(t + s)] 


8 (t+) Py i ee 
ae 3 + 373 


4t? 3 
ata +2072 +te += 


4.2.4 Brownian bridge 


The processes that we studied so far in this chapter were all defined for values 
of the variable t in the interval [0, 00). However, an interesting process that 
is based on a standard Brownian motion, {B(t),t > 0}, and that has been 
the subject of many research papers in the last 20 years or so, is defined 
for t € [0,1] only. Moreover, it is a conditional diffusion process, because we 
suppose that B(1) = 0. Since it is as if the process thus obtained were tied at 
both ends, it is sometimes called the tied Wiener process, but most often the 
expression Brownian bridge is used. 
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Definition 4.2.5. Let {B(t),t > 0} be a standard Brownian motion. The 
conditional stochastic process {Z(t),0 <t <1}, where 


Z(t) = B(t) | {B(1) = 0} (4.114) 
is called a Brownian bridge. 


Remark. We deduce from the properties of the Brownian motion that the 
Brownian bridge is a Gaussian diffusion process (thus, it is also Markovian). 


Suppose that the standard Brownian motion {B(t),t > 0} starts in fact 
from B(0) = bo and that B(s) = b,. Then, using the formula (which follows 
from the (independent and) stationary increments of the Brownian motion) 


fB(t),B(s)(0,5s) face) (0) fB(s)— Bt) (bs — 5) 


f =p,)(b) = Se eee (4,115) 
B(t)|{B(s)=b.} (0) Faia) Fass) ( 

for 0 < t < s, we find that 

B(t) | {B(0) = bo, B(s) = bs} ~ Nn (7 + wee ¥ te (0,8) 


(4.116) 
Thus, when 9 = 0, s = 1, and b, = 0, we obtain that 
B(t) | {B(0) = 0, B(1) = 0} ~ N(0,t(1—t)) Vte (0,1) (4.117) 
so that 
E(Z(t)]} =0 Vte (0,1) (4.118) 
and 
E(Z?(t)] = V[Z(t)] =t(1—t) Vt (0,1) (4.119) 


With the help of the formulas (4.116), (4.118), and (4.119), we calculate 
the autocovariance function of the Brownian bridge, for 0 < t <7 < 1, as 
follows: 


Ca(t,T) = = CovlZ(), Z(r)| = E[Z(t)Z(r)] — 0 x 0 
ElE eg [2(7)B[Z(t) | Z(7)]] 
) 


t 
(2 (r)22(r)| = Sra) 
T 
=t(1-71) Sereda (4.120) 
In general, we have 


Cz(t,r) = min{t,r}—tr if t,r € (0,1) (4.121) 
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We also deduce from the formula (4.116) that 
E|Z(t+) — Z(t) | Z(t) = 2] = -—— (4.122) 
and 
E((Z(t + €) — Z(t))? | Z(t) = 2] =e + of) (4.123) 


It follows that the infinitesimal parameters of the Brownian bridge are given 
by (see p. 63) 


zx 
ee eee 4124 
a ie Ce a a 


and 


teats (4.125) 
€j0 € 
forz €RandO<t<1. 


Remarks. i) The Brownian bridge thus has the same infinitesimal variance as 
the standard Brownian motion. However, its infinitesimal mean depends on ¢ 
and tends to infinity (in absolute value if z # 0) as t increases to 1. 


ii) Since m(z;t) is a function of t, the Brownian bridge is not a time- 
homogeneous process. 


As we did in the case of the Brownian motion, we can use the fact that 
the Brownian bridge is a Gaussian process to give a second definition of this 
process as being a Gaussian process with zero mean and whose autocovariance 
function is given by the formula (4.121). 


Proposition 4.2.6. Let {B(t),t > 0} be a standard Brownian motion. The 
stochastic process {Z;(t),0 <t <1} defined by 


21(t) = B(t) — tB(1) (4.126) 
is a Brownian bridge. 


Proof. We can assert that {Z;(t),0 < t < 1} is a Gaussian process. Moreover, 
we calculate 


E{Z,(t)| = E[B(t)| — tE|B(1)] =0-tx0=0 (4.127) 
and then, 


Coy[Z1(t), Z1(7)] = E[Z,(t)Z1(7)] = E[(B(t) — tB(1))(B(r) - 7B(1))] 
= E[B(t)B(r)] — 7 E[B(t)B(1)] — t E[B(1)B(r7)] 
+ trE|B?(1)| 
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=min{t,r}-7rxt—-—tx7+trxl 
= min{t,r} —tr (4.128) 


Since the mean and the autocovariance function of the Gaussian process 
{Z,(t),0 < t < 1} are identical to those of the Brownian bridge, we can 
conclude that it is indeed a Brownian bridge. 0 


Remark. In the same way, we can show that the process {Zo(t),0 <t < 1} 
defined by 


Z(t) =(1-t)B (4) for0<t<1 (4.129) 
and Z2(1) = 0 is a Brownian bridge as well. 


4.2.5 The Ornstein—Uhlenbeck process 


As we mentioned in Section 4.1, the use of a Wiener process to model the 
displacement of a particle can be criticized. Indeed, for small values of the 
variable t, the Wiener process is not appropriate to represent the average ve- 
locity of the particle in the interval [0, t]. Moreover, the instantaneous velocity 
cannot be calculated, because the Brownian motion is nowhere differentiable. 

To remedy this problem, in 1930 Uhlenbeck? and Ornstein® proposed a 
model in which they supposed that it is the velocity of the particle that is 
influenced, in part, by the shocks with the neighboring particles. The velocity 
also depends on the frictional resistance of the surrounding medium. The effect 
of this resistance is proportional to the velocity. 

As in the case of the geometric Brownian motion, we can define the 
Ornstein-Uhlenbeck (O.-U.) process from a Wiener process. Let {B(t),t > 0} 
be a standard Brownian motion. We set 

2 p2at 


U(t) = e-“B (: e ) for t > 0 (4.130) 


2a 


where a is a positive constant. It is thus a particular case of the transformation 
X(t) = g(t) B( F(t) (4.131) 


where f(t) is a nonnegative, continuous, and strictly increasing function, for 
t > 0, and g(€) is a (real) continuous function. Indeed, the exponential function 


? George Eugene Uhlenbeck, 1900-1988, was born in Indonesia and died in the 
United States. He was a physicist and mathematician whose family, coming from 
the Netherlands, returned there when he was six years old. His main research 
subject was statistical physics. He also worked on quantum mechanics and wrote 
two important papers on Brownian motion. 

3 Leonard Salomon Ornstein, 1880-1941, was born and died in the Netherlands. 
He was a physicist who worked on quantum mechanics. He applied statistical 
methods to problems in theoretical physics. 
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is continuous and (o7/2a)e2 is nonnegative and strictly increasing, because 
a > 0. For any transformation of this type, the following proposition can be 
shown. 


Proposition 4.2.7. Under the conditions mentioned above, the stochastic 
process {X(t),t > 0}, where X(t) is defined in (4.131), is a Gaussian process, 
whose infinitesimal parameters are given by 
sgn g'(t) oa) a ne ! 
m(x;t) = 0 xz and v(x;t) = 9*(t)f'(t) (4.132) 


forz ER andt>0. 


Remark. The process {X(t),t > 0} is Gaussian, because the transformation 
of the Brownian motion is only with respect to the variable t (see p. 179). 
That is, we only change the time scale. 


From the formulas in (4.132), we calculate 


—ae~% 


m(z;t) = mee 


2 
z=-axr and v(r;t)= (ent)? P20 =o? (4.133) 


Definition 4.2.6. The stochastic process {U(t),t > 0} whose infinitesimal 
parameters are given by my(u;t) = —au, for allt > 0, where a > 0, and 
vy (u;t) = 0? is called an Ornstein—Uhlenbeck process. 


Remarks. i) We see that the Wiener process can be considered as the particular 
case of the O.-U. process obtained by taking the limit as a decreases to zero. 
Conversely, if {U(t),t > 0} is an O.-U. process and if we set B(0) = 0 and 


g2 \1/? 1 2at 
Bt) =(— — — 4.134 
(t) (sa) u| n (35 )| fort >0 ( ) 


then {B(t),¢t > 0} is a standard Brownian motion. 


ii) We deduce from the definition, given in (4.130), of an O.-U. process in 
terms of {B(t),t > 0} that {U(t),t > 0} is a Markovian process. It is also a 
diffusion process. 


iii) Note that the initial value of an OU. process, as defined above, is a 
random variable, since 


U(0) = &B (<=) = B (5) ~N (0 =) (4.135) 


We can arrange things so that U(0) = uo, a constant (see Ex. 4.2.3). 


We calculate the mean and the variance of the O.-U. process from (4.130). 
We have 
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o2e2ut 
E(U(t)| = E own (<=) =0 (4.136) 


and 


2 ,2at 2 ,2at 2 
V[U(t)] = e2'v LB (2 c J ST a = fort >0 (4.137) 


2a 2a 
Next, using the formula Cov[B(s), B(t)] = min{s,t}, we calculate 
e2at oze2alt+s) 
Cov[U (t), U(t + s)] =e7 Me“ 89) a Gs ).B (-— a) 
2a 
2,2at as 
=e72ate-aeF T  _ for s,t>0 (4.138) 
2a oR 


Given that the mean of the process is a constant and that its autocovariance 
function Cy(t,t + s) does not depend on t, we can assert that the O.-U. 
process is a wide-sense stationary process (see p. 53). Moreover, since it is a 
Gaussian process, we can state the following proposition. 


Proposition 4.2.8. The Ornstein—Uhlenbeck process is a strict-sense station- 
ary process. 


Contrary to the Brownian motion, the increments of the O.-U. process 
are not independent. Indeed, we deduce from the formula (4.138) [and from 
(4.136)] that 


Cov[U(t + s) — U(t), U(t) — U(0)] 


E\{UG@+4+s) (t)}{U(t) (0)}} -O0x0 
_ BIE ‘ Mae E[U(t + s)U(0)] — E[U2(t)] + E[U(t)U(0)] 
= us (eo — em alt+s) yg - #0 (4.139) 


(if s > 0) so that we can assert that the random variables U(t + s) — U(t) and 
U(t) — U(0) are not independent for all s,t > 0. 

However, as the O.-U. process is (strict-sense) stationary, its increments 
are stationary. To check this assertion, we use the fact that the O.-U. process 
is Gaussian, which implies that the random variable U(t) — U(s), where 0 < 
s < t, has a Gaussian distribution, with zero mean, and whose variance is 
given by 


V[U(t) — U(s)] = VIU(t)] + V[U(s)] — 2 Cov[U(t), U(s)] 
Z (< S eS (4.140) 


2a 2a 


Now, this variance is identical to that of the variable U(t + 7) - U(s +7), 
which also has a Gaussian distribution with zero mean, from which we can 
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conclude that the random variables U(t) - U(s) and U(t +7) —U(s +7) are 
identically distributed for all r > 0. 


Let us now return to the problem of modeling the displacement of a par- 
ticle, for which we proposed to use a Brownian motion. If we suppose that 
U(t) is the velocity of the particle, then we may write that its position X(t) 


at time t is given by 
t 
a / U(s) ds (4.141) 
0 


We can therefore assert that X(t) —X(0) has a Gaussian distribution (because 
the O.-U. process is Gaussian), with mean [see (4.136)] 


E|X(t) — X(0)] = E if U(s) as “ i: E[U(s)] ds = i Ods=0 (4.142) 


and variance [see (4.138)] 


V[X(t) — X(0)] = E[(X(t) — X(0))?] — 0? 


= 8] f vis) as [ u(r) ar| 

= eeu) ds dr 

“hh aoe | ds dr 

=< ; te (= Past fe ero ash dr 


oft 
eee _ par _ ,a(r-t) 
Fal i {2 e€ e€ } dt 
z ~att 4.143 
eg (at -l+e ) (4.143) 
From the series expansion of e~®: 
1 1 
“ot — 1 at+ 5 (at)? -  (at)® oe (4.144) 


we may write that 


2 
2 +? if t is small 
a 
V[X(t) — X(0)] ~ (4.145) 
ee 
—zt ift is large 
a 


Thus, the variance of the integral of the O.—U. process tends to that of a 
Brownian motion when ¢t tends to infinity. However, for small values of t, the 
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variance of the integral is proportional to the square of the time ¢ elapsed 
since the initial time instant. Now, this result is more realistic than assuming 
that the variance is always proportional to t. Indeed, let X ~ N(0,0%). By 
symmetry, we may write that 


et 2/(20%) 
E\|X|}=2 f —=—e* "x? der 
Ixy =2f 
1 2 2 a V2 
sro oze* /(20x) = ao 4,146 
V2n0x ) Jr “ 


Since the standard deviation of X (t)—X (0) is proportional to ¢ for small values 
of t, we deduce from (4.146) that the order of magnitude of the velocity of the 
particle in a small interval of length 6, from the initial time, is a constant: 


V2 (a/V2a)5 ee a pe aa SO (4.147) 


Ji 6 Jno 
whereas the order of magnitude of this velocity tends to infinity in the case 
of the Brownian motion (see p. 176). 


The Kolmogorov forward equation corresponding to the O.—U. process is 
the following: 


Op fa) a? Op 
pel ieee ue cee 4,148 
Ot arm p) + 2 du? ( ) 
The solution that satisfies the initial condition 
lim p(u, uo; t, to) = 6(u — uo) (4.149) 
thto 
is 
1 1 : 
p(u, uo; t, to) = FS OP 5 (u— pov) (4.150) 
V 2k06y Tou 
for uo,u € R and t > to > 0, where 
Pe) 
Bou := Uo e* &-) and 024 := Balt sig eet) (4.151) 


That is, U(t) | {U(to) = uo} ~ N(uovu, cy): 

We deduce from what precedes that the stochastic process defined from 
a standard Brownian motion in (4.130) is actually the stationary version of 
the Ornstein—-Uhlenbeck process, obtained by taking the limit as ¢ tends to 
infinity of the solution above. Indeed, we have 


2 
; , o 
jim Lov =0 and jim, on a (4.152) 
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as in (4.136) and (4.137). 


Remark. The solution (4.150) enables us to state that the O.-U. process is 
time-homogeneous. However, this version of the process is not even wide-sense 
stationary, since its mean depends on t. 


Example 4.2.3. For the initial value of the O.-U. process {U(t),t > 0} to be 
deterministic, we can set, when a = 1 and o? = 2, 


U(t) = uo + e ' B(e**) — B(1) 
We then have 
E[U(8)] = wo +e B[B(c%)] — BIB(1)] = wo 
and, for all t > 0, 


V[U(t)] = Vie *B(e*) — B()] 
= e~*V[B(e**)| + V[B(1)] — 2e7'Cov[B(e”4), B(1)] 
= 7 7t ot 44 — 2e7t min{e**, 1} 
=2-2e* x1=2(1—-e) 


4.2.6 The Bessel process 


Let {B,(t),t > 0}, for k = 1,...,n, be independent standard Brownian 
motions. We set 


X(t) = Be(t)+...+ B2(t) Vt>0 (4.153) 


The random variable X(t) can be interpreted as the square of the distance 
from the origin of a standard Brownian motion in n dimensions. The following 
proposition can be shown. 


Proposition 4.2.9. The stochastic process {X(t),t > 0} is a diffusion process 
whose infinitesimal parameters are given by 


m(z;t)=n and v(r;t)=4r forx >0 andt>0 (4.154) 


Remark. Note that the infinitesimal parameters of the process do not depend 
on the variable t. 


We now define the process {Y(t),t > 0} by 
Y(t) = g[X(t)] = X*/2(t) fort >0 (4.155) 


Since the transformation g(x) = 2x/? is strictly increasing, and its second 
derivative g(x) = —42~8/2 exists and is continuous for x € (0,00), we can 
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use the formulas in (4.35) to calculate the infinitesimal parameters of the 
diffusion process {Y(t),t > 0}: 


1 Ay? 1 n-1 
eae Lies Ge Se iy Gee ene 4.156 
n(s)+ 2 ( a) 2y oe) 


2 
Wits (sa) 24 (4.157) 


and 


It can be shown that this process is indeed Markovian. Moreover, we can 
generalize the definition of the process {Y(t),¢ > 0} by replacing n by a real 
parameter a > 0. 


Definition 4.2.7. The diffusion process {Y (t),t > 0} whose state space is the 
interval [0,00) and whose infinitesimal mean and variance are given, respec- 
tively, by 


and v(y;t)=1 fort >0 andy>0 (4.158) 


m(y;t) = = 
is called a Bessel* process of dimension a > 0. 


Remarks. i) The term dimension used for the parameter a comes from the 
interpretation of the process when a=neéN. 

ii) When a = 1, the infinitesimal parameters of the process are the same 
as those of the standard Brownian motion. However, we can represent it as 
follows: 


Y(t) =|B(t)| fort >0 (4.159) 


where {B(t),t > 0} is a standard Brownian motion. It is as if the origin were 
a reflecting boundary for {B(t),t > 0}. 

The conditional transition density function of the Bessel process satisfies 
the Kolmogorov forward equation 


> 2 
S/O SE a (4.160) 
Ot 2 oOy\y 2 Oy? 
We can check that 
i ae ea yety? yoy 
poegred ta, _Yory | (You 4.161 
p(y, Yo; t, to = 0) ; (+) yexp { “Teale ee ( =) ( ) 


4 See p. 114. 
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for yo, y, and t > 0, where 
a 


= —-—i1 4.162 
vag (4.162) 


and I,(-) is a modified Bessel function of the first kind (of order v), defined 
by (see p. 375 of Ref. [1]) 


=e): 2 aw FRAT) oF ms ; < —; oe) 


Remark. The quantity v defined in (4.162) is called the index of the Bessel 
process. 


Definition 4.2.8. The diffusion process {X(t),t > 0} defined by 
X(t)=Y7(t) fort >0 (4.164) 


is called a squared Bessel process of dimension a > 0. Its infinitesimal 
parameters are given by 


m(z;t)=a and v(a;t)=4r fort >0 andzx>0 (4.165) 


Remarks. i) We deduce from the representation of the process {X(t),t > 0} 
given in (4.153) that if {X(t),¢ > 0} and {X2(t),t > 0} are two independent 
squared Bessel processes, of dimensions a; = n, and ag = ng, respectively, 
then the process {S(t),t > 0}, where 


S(t) := X,(t) + X2(t) fort >0 (4.166) 


is also a squared Bessel process, of dimension @ := n, + ng. Actually, this 
result is valid for all ay > 0 and a2 > 0, and not only when a; and ay are 
integers. 

ii) The function p(x, x0;t, to = 0) of the squared Bessel process of dimension 
a is given by 


Peet = Oe (2) exp \-2 ss = L (=) (4.167) 


t \ zo 2t t 


for x9, x, and ¢t > 0, where v = (a/2) —1. 


A diffusion process used in financial mathematics to model the variations 
of interest rates, and which can be expressed in terms of a squared Bessel 
process, is named the Cox—Ingersoll—Ross®- (CIR) process (see Ref. [4]). 
We set 


® John C. Cox and Stephen A. Ross are professors at the MIT Sloan School of 
Management. Jonathan E. Ingersoll, Jr. is a professor at the Yale School of Man- 
agement. 
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o2 


R(t) = eX ( 7 


(eo — 1) for t > 0 (4.168) 


where b € R and o > 0 are constants, and {X(t),t > 0} is a squared Bessel 
process of dimension a = 4a/o? (with a@ > 0). We find that the infinitesimal 
parameters of the process {R(t),t > 0} are given by 


m(r;t)=a—br and v(r;t)=o?r fort >0 andr >0 (4.169) 


From the formula (4.167), we find that 


bt be \ ¥/2 bt forget 


2k(t) \ ro 2k(t) k(t) 
(4.170) 
for ro, r, and t > 0, where 
2 
oO" 1 bt 2a 
pes _ _ ae | 4.171 
k(t) a (e%—1) and v = ( ) 


4.3 White noise 


In Section 2.4, we defined the stochastic process called white noise as being 
a process (that we now denote by) {X(t),t > 0} with zero mean and whose 
autocovariance function is of the form 


Cx (ti, te) = q(t1)d(te — ty) (4.172) 


where q(t;) is a positive function and 46(-) is the Dirac delta function (see 
p. 63). 

When q(t1) = 0”, the function Cx (t1, tz) is the second mixed derivative of 
the autocovariance function Cy (t1,t2) = 0? min{t,,t2} of a Brownian motion 
(without drift and) with diffusion coefficient o?. Indeed, we have 


6) us 0 ifty < te 
aay lw (tt) = {o Loe (4.173) 
so that 
Jee ec, to) = cae es = tz) = o75(ty = to) = o75(te ant ti) (4.174) 
Ot Ote , Oty 


where u(-) is the Heaviside function (see p. 11). 


Definition 4.3.1. The (generalized) stochastic process {X (t),t > 0} with zero 
mean and autocovariance function 


Cx (t1,t2) = 076(t2 — t1) (4.175) 


is called a Gaussian white noise (or white Gaussian noise). 
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Remarks. i) Since E[X(t)] = 0, we also have 


Rx (ty, t2) = 075(te — ty) (4.176) 


The second mixed derivative wea Fy (ti, t2) is called the generalized mean- 
square derivative of the stochastic process {Y(t),t > 0}. We say that the 
process {X(t),¢ > 0} having the autocorrelation function aa Ry (tr, ta) is 
a generalized stochastic process. 

ii) A white noise (Gaussian or not) is such that the random variables X(t,) 
and X(t2) are uncorrelated if t; # ta. If the variables X(t:) and X(t2) are 
independent, the expression strict white noise is used to designate the corre- 
sponding process. 


Now, we mentioned that the Brownian motion is nowhere differentiable. 
Consider, however, the process {X(t),t > 0} defined (symbolically) by 


yi si Pe) 


4.177 
€l0 € ( ) 


where € > 0 is a constant and {W(t),t > 0} is a Brownian motion with 
coefficients 4 = 0 and o? > 0. Assume that we can interchange the limit and 
the mathematical expectation. Then 


W(t+e)- mn 
€ 


=lim0 =0 (4.178) 


Ex @) = ane | €10 


so that 


Cxlti,ts) = lim B ( +e)- Win) (we +e) — wie) 
€10 


€ € 


= lim a{E [W(t +e)W(te +€)] — E[W(ti + )W(t2)] 


- B[W(e)W(e +0 +E) } (4.179) 


Suppose first that t; < t2. Then, for an € small enough, we will have that 
t) +€ < to. It follows that 


2 2 
_ 6 _ 6 
Cx (ti, tz) = sey {(t1 +e) —(ti te) —tr t+ti} = as. 0=0 (4.180) 
When ¢, = tg, we obtain 


2 
~ 2 
Cx (t, tz) = lim > {(i + €) = t —t, +t} 
elO € 
2 o2 


= lim a ¢ = lim — = 00 (4.181) 
el0 € elO € 
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We can therefore write that 
Cx (t, t2) = 076(tz — t) (4.182) 


Thus, we interpret a Gaussian white noise as being, in a certain way, the 
derivative of a Brownian motion. From now on, we will denote the Gaussian 
white noise by {dW(t),t > 0}. We also have that dW(t) = W(t) dt. In 
engineering, the notation dW (t)/dt = e(t) is often used. 


Even though the Brownian motion is not differentiable, we can use the 
notion of a generalized derivative, which is defined as follows. 


Definition 4.3.2. Let f be a function whose derivative exists and is continu- 
ous in the interval [0,t}. The generalized derivative of the function W(t), 
where {W(t),t > 0} is a Brownian motion, is given by 


| f(s)W'(s) ds = f(t)W(t) - 1 W(s)f"(s) ds (4.183) 
0 0 


Remarks. i) This definition of the generalized derivative of W(t) is actually a 
more rigorous way of defining a Gaussian white noise {dW (t),t > 0}. 


ii) For a deterministic function g, its generalized derivative is defined by 


oO oo 
J seas) as == [af s)s'(s) as (4.184) 
0 0 
where we assume that f(t) =0 for all ¢ € [a,b], with a > 0 and b < ~. 


The formula (4.183) leads us to the notion of a stochastic integral. 


Definition 4.3.3. Let f be a function whose derivative exists and is contin- 
uous in the interval [a,b], where a > 0, and let {W(t),t > 0} be a Wiener 
process. We define the stochastic integral is f(t) dW(t) by 


b b 
i f(t) dW (t) = f(b)W(b) — f(@)W(a) — | W(t) df(t) (4.185) 


Remarks. i) We can also define a stochastic integral as follows: 


b n 

[ toaww= lim S> f(ti-a)[W(ts) — W(ti-a)]. (4.186) 
a moo ix 
ana ie 7 tia} | " 


where a = tp < ty <...<t, =b is a partition of [a, Dj. 


ii) The definition of ri f(t) dW(¢) follows from the formula (4.177) as well, 
by setting 


210 4 Diffusion Processes 


[ F(0) aW(t) = lim f ° F(t) jean dt (4.187) 
Indeed, using the formula 
W(t+e)-W(t)_ d (1 se 


and integrating by parts, we obtain 


i "FW awit) = in { fcr Wis) as 


Est a 


~ | “F) (- wie) as) at\ ete) 


Finally, since W(t) is a continuous function, we have (making use of 
l’Hospital’s® rule) 
t+e t+e 


1 d 
im — = lim — W(t) (4.190 
lim =a) W(s) ds lim qe: W(s) ds = lim W(t+e)=Wi(t) ( ) 


from which we retrieve the formula (4.185). 


Properties. i) From the formula (4.185), we can assert that a stochastic 
integral has a Gaussian distribution, because it is a linear combination of 
Gaussian random variables. 


ii) We have 


b 
= / W(t) df(t) 


(4.191) 


b 
E| [ft awe) = f(b)E[W(b)] - fla)E(W(a 


where we assumed that we can interchange the mathematical expectation and 
the integral. 


iit) To calculate the variance of a stochastic integral, we can use the formula 
(4.186) and the fact that the increments of the Brownian motion are indepen- 
dent and stationary. We have 


nr 


V |S> f(ti—1)[W (te) — W(te-1) 


t=1 


?(t;_1) V[W (ti) — W(ti-1)] 


Mee li it 


f?(ti1)V [W(t — ti-1)] 


1 


a 
ll 


® Guillaume Francois Antoine de PHospital, 1661-1704, was born and died in 
France. He published the first textbook on differential calculus, in which the 
rule that bears his name can be found. 
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n 


=S5 P(ter)er(ti—ti-1) (4-192) 


i=l 


It follows (interchanging the limit and the calculation of the variance) that 


b n 
V i f(t) avo] = lim SS P(ti-r)o7 (ts — ty-1) 
ss nm — 00 i=l 
eA % tia} ! : 
=o" il f?(t) dt (4.193) 


iv) We can generalize the preceding formula as follows: 


0 ifa<b<c<d 
E 


b d 
i f(t) awit) / g(t) awio| 7 : 
ri a f f(t)g(t) dtifa=c<b<d 


(4.194) 


where g is a function whose derivative exists and is continuous in the interval 


[e, d]. 


Let {X(t),t > 0} be a continuous-time and continuous-state stochastic 
process whose infinitesimal parameters are m(x;t) and v(z;t). This process 
can be represented in the following way (see Ref. [16], for instance): 


t t 
X(t) = X(0) +f m[X(s); s] as+ f v'/?(X(s); s] dB(s) (4.195) 


where {B(t),¢ > 0} is a standard Brownian motion. It follows that X(t) is a 
solution of the stochastic differential equation 


dX (t) = m[X(t); t] dé + v'/?[X(t);t] dB(t) (4.196) 

Under the condition X(t) = x, the equation above becomes 
dX(t) = m(2;t) dt + v'/?(a;t) dB(t) (4.197) 
<> X(t) =m(a;t) + 01/2 (a;t) B(t) (4.198) 


(with the notation X(t) = X(t). 
We can consider stochastic differential equations in n dimensions. A useful 


result is given in the following proposition. 


Proposition 4.3.1. Let {X(t),t > 0} be an n-dimensional stochastic process 
defined by 
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dX(t) = (AX(t) +a) dt +N'/? dB(t) (4.199) 


where {B(t),t > 0} is an n-dimensional standard Brownian motion, A is a 
square matrix of order n, a is an n-dimensional vector, and N!/2 is a positive 
definite square matrix of order n. Then, given that X(t) = x, we may write 
that 


X(t) ~ N(m(t), K(t)) fort > to (4.200) 
where 
m(t) := &(t) (x + i O*(u) a iu) (4.201) 
and 


K(t) := S(t) ( i S~*(u)N[S-*(u)] au) ©’ (t) (4.202) 


to 


where the symbol prime denotes the transpose of the matrix, and the function 
P(t) is given by 


4, (t — to)” 
__ ,A(t-to) _ n\t — to)” 
G(t) :=e =S°A = (4.203) 
n=0 
Remarks. i) A matrix M = (i,j); 5.1... » i8 positive definite if 
n n 
c’Mc = Ss S> ccjm,j > 0 (4.204) 
i=1 j=l 
for any vector c! := (c1,... ,Cn) that is not the null vector (0,... ,0). 


ii) When n = 1, the formulas for the mean and the variance become 


t + a(t — to) if A=0 
m(t) = (4.205) 
GY) A(t-to)__ @ ; 
(« + “) e = if A#0 
and 
N(t = to) if A=0 
K(t) = (4.206) 
N_ (,2A(t~to) : 
i (c af 1) if AZO 


iii) We can generalize the proposition to the case where A = A(t), a = a(t), 
and N!/2 = N¥/2(). The function @(t) is then obtained by solving the matrix 
differential equation 


d 
Felt) = AH) (4.207) 


with the initial condition O(t9) = I, (the identity matrix of order n). 
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Example 4.3.1. The Brownian motion {X(t),¢ > 0} with drift coefficient up # 
0 and diffusion coefficient 7? (> 0) is defined by the stochastic differential 
equation 

dX (t) = pdt + od B(t) 


According to the second remark above, the random variable X(t), given that 
X(0) = 2, has a Gaussian distribution with parameters 


Hx(e) =™m(t)=a2+ pt and ox) = K(t) =o7t 


which does correspond to the results mentioned in Subsection 4.2.2. 


In the case of the Ornstein—Uhlenbeck process, {U(t),t > 0}, we have the 
following stochastic differential equation: 


dU(t) = ~a U(t) dt +0 dB(t) 


so that A = —a, a = 0, and N!/2 = o. It follows that the random variable 
Y(t) := U(t) | {U(to) = uo} has a Gaussian distribution whose parameters 
are (see p. 203) 


2 
Hy (t) = uo e*) and oF) = = ere = 1) 


Example 4.3.2. Consider now the two-dimensional diffusion process X(t) = 
(Z(t), Y(t)) defined by the system of equations 


dZ(t) =Y(t) dt 
dY (t) = ys dt + cdB(t) 


That is, {Y(t),t > 0} is a Brownian motion with drift coefficient  (€ R) and 
diffusion coefficient o? (> 0), and {Z(t),t > 0} is its integral. Suppose that 
(Z(0), Y(0)) = (z,y). We may write that 


_ {01 _ 10 1/2 __ |90 
a=([5a)> a=[if. and N = [50 


Let us first calculate the function &(t). We have 
2_ |01;|01;_ {00 
Seal ioe | a foo 


@(t) =p + At = E | 


It follows that 


01 
so that 
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[ow ce [ ea ie ae 


which implies that 


We then have 


z+yt+ gut? 
y + pt 


3 
oh 
wo 
Il 
-_ 
or 
Roo 
Cena 
reo 
ce 
Ce kh 
te 
+ 
_— 
{ 
Hod 
BE 
oh 
N 
ease 
SS 
{i 


Next, we have 


and 


Leromerora- fee] ea] [23]} « 


- at /3 | 


~ | -o7t?/2 ot 


from which we calculate 
K(t) = 1t] [ 07t3/3 —o7t?/2] [10] _ [07t3/3 o7t?/2 
~ 101] |-07t?/2 ot tl] |ot?/2 ot 


Note that these results agree with those mentioned in the case of the three- 
dimensional process (Y(t), Z(t), D(t)) (see p. 196). 


4.4 First-passage problems 


Let Ty be the random variable that designates the time needed for a standard 
Brownian motion to go from B(0) = 0 to B(t) = d # 0. Symbolically, we 
write 


Tq := min{t > 0: Bit) = d} (4.208) 


Remark. To be more rigorous, we should write that Ty is the infimum (rather 
than the minimum) of the positive values of t for which B(t) = d. Indeed, in 
the case of the standard Brownian motion, we can show [see Eq. (4.218)] that 
the probability that the process will eventually hit the boundary at d is equal 
to 1, for any d € R. However, in other cases, it is not certain that the process 
will hit the boundary at d. If the stochastic process does not hit the boundary, 
the set whose minimum is to be found is then empty and this minimum does 
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not exist. On the other hand, the infimum of the (empty) set in question is, 
by definition, equal to infinity. 


Suppose first that d is a positive constant. We can obtain the distribution 
of the random variable Ty by using the fact that the distribution of B(t) is 
known, and by conditioning on the possible values of Ty. We have 


P[B(t) > d] = P[B(t) > d| Ta < t]P[Tu < t] + PIB(t) > d| Ty > t)P[Ta > tt] 
(4.209) 


We deduce from the continuity of the process {B(t),¢ > 0} that 
P[B(t) > d| Ty >t] =0 (4.210) 
Moreover, if Ty < t, then there exists a tg € (0,t) such that B(to) = d. Since 
Bit) | {B(to) =d} ~ N(d,t—to) for t > to (4.211) 


we may write (by symmetry) that 
1 
P(B(t) > d| Ta <t}= 3 (4.212) 


Finally, P[B(t) > d | Ta = t] = 1. However, as Ty is a continuous random 
variable (which implies that P[Ty = t] = 0), we may conclude that 


PiTy < t] = 2|1- a(d/vi)| , where ®(x) := PIN(0,1) <a] (4.213) 


Remarks. i) The preceding formula is called the reflection principle for the 
Brownian motion. 


ii) We also deduce from the continuity of the Wiener process that 


P (Eas BO >d(> 0) SPs t= 2 [2 - o(a/vi)| (4.214) 
sss 

That is, the probability that the process takes on a value greater than or equal 

to d > 0 in the interval (0, t] is the same as the probability that it reaches the 

boundary at d not later than at time t. 


iii) By symmetry, Tg and T_g are identically distributed random variables. It 
follows that 


PIT, <#)=2 E = #(\d\/v2)| vd#0 (4.215) 


iv) The density function of Ty is obtained by differentiating the function above: 


ad \d|/vi 4 P 
t) = — ja(1- ae? 
fr, (t) ( oe Wor z 


dt 
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re ee a) (__lal 
Tag PL of 287 


2 
= exp {-} fort > 0 (4.216) 


This density is a particular case of the inverse Gaussian or Wald’ distribution. 


Remark. The density function of a random variable X having an inverse Gaus- 
sian distribution with parameters p: > 0 and 2 > 0 is given by 


_ 4 )2 
fx (tind) = ee exp { ACS HP | Va>0 (4.217) 


The distribution of Ty above is thus, in fact, obtained by setting \ = d? and 
by taking the limit as the parameter yz tends to infinity. 


Note that from (4.215), we deduce that 
1 
P[Ta < oo] = jim P[Ta < t] = 2[1 — (0)] =2 ¢ - 5) =1 (4.218) 


Therefore, the standard Brownian motion is certain to eventually reach any 
real value d. However, the formula (4.216) implies that 


rirj- f — Me yes (4.219) 
Oe Jo fame Bef 
because 

1 d? 1 


That is, the function that we integrate behaves like Go 


Given that 


as t tends to infinity. 


oO 
/ t'/? dt= oo Vto>0 (4.221) 
t 


10) 
we must conclude that the mathematical expectation of the random variable 
Tq is indeed infinite. 
We also deduce from the formula (4.215) that 


Pld?T, < 1] =2 E _@ (./ vir) | =2 E = (ial/vé)| (4.222) 


” Abraham Wald, 1902-1950, was born in Kolozsvar, Hungary (now Cluj, in Roma- 
nia), and died (in a plane crash) in India. He first worked on geometry and then 
on econometrics. His main contributions were, however, to statistics, notably to 
sequential analysis. 
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Thus, the random variables Ty and Tj := d?T; are identically distributed. 
Finally, let X := 1/B?(1). Since B(1) has a Gaussian N(0, 1) distribution, 
we calculate, for x > 0, 


P[X <a] = P[B?(1) > 1/2] =1- P[-1/Vz < B(l) <1/Ve] 
=1-(2 6(1//z) —1) =2(1-6(1/V2)) (4.223) 


from which we can assert that T; and X are also identically distributed ran- 
dom variables. 


Remark. The square of a standard Gaussian random variable Z has a chi- 
square distribution with 1 degree of freedom, which is also a particular gamma 
distribution: Z? ~ G(1/2,1/2). The random variable T; is thus the reciprocal 
of a variable having a gamma distribution. 


A much more general technique to obtain the distribution of a first-passage 
time T for an arbitrary diffusion process consists in trying to solve the Kol- 
mogorov backward equation satisfied by the density function of this random 
variable, under the appropriate conditions. 

Let {X(t),t > 0} be a time-homogeneous diffusion process whose infinites- 
imal parameters are m(x) and v(z). Its conditional transition density function, 
p(x, 20;t, to), satisfies 


Op dp 1 ap 
a pac Sie et —_~—=0 4,224 
Bto + m(ro) a + 9°(%0) a3 ( ) 


If tg = 0, we can use the fact that the process is time-homogeneous, so that 


6) fe) 0 
Beg Pm 705 ts to) = Beg Plt B03 # —to) = — By P(® 20st ~ to) (4.225) 


to write that 


5°00) gen + mo) <> a as (4.226) 
Let p(t; zo) be the probability density function of 
Tod (= Te,a(Zo)) = min{t > 0: X(t) ¢ (c,d) | X(0) = 20 € [e,d]} (4.227) 
That is, 
p(t; x9) dt := P[Tea € (t,t + dt] | X(0) = zo € [c,d] } (4.228) 


We find that the function p(t;x) satisfies the partial differential equation 
(4.226). Moreover, since the coefficients m(zg) and v(zp) do not depend on 
t, we can take the Laplace transform of the equation (with respect to t) to 
reduce it to an ordinary differential equation. Let 
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L(x9; a) := / e p(t; x0) dt (4.229) 
0 


where a is a real positive constant. 


Remark. The Laplace transform of the density function of the continuous and 
nonnegative random variable X is the moment-generating function Mx(—a) 
of this random variable (see p. 19). 


We can check that the function L(x9; @) satisfies the following differential 
equation: 


1 OFL OL 
sal a, —= 4.230 
5U (x0) dx2 + mx) 0x0 aL ( ) 


This equation is valid for c < 29 < d. Indeed, we have 
—at a ~—at oo ici —at 
ee — p(t; 29) dt = e~ p(t; x0) |6 +a e "p(t; x9) dt 
0 ot 0 
=0+aL=alL (4.231) 


because p(0; x9) = 0 if zo # cor d. Since T,¢ = 0 if ro = c or d, the boundary 
conditions are 


L(ao;a) = E [eo 


X(0)=20] =1 ifzg=cord (4.232) 


Once we have found the function L(zo;@), we must invert the Laplace 
transform to obtain the density function of T.,a. 


Example 4.4.1. Let {X(t),t > 0} be a Wiener process whose drift and dif 
fusion coefficients are ys and o?, respectively. We must solve the following 
differential equation: 
o? &L OL _ 
2 O28 *Sr5 


The general solution of this equation is given by 


aL 


L293) = cpe797 ©) +4 cpetor2(@) (4.233) 


where c; and cz are constants, and 
1 1 
T1(Q) = 2 (-n — Veet 2007) and re(a) := 52 (-n + fp? + 2007) 
The solution that satisfies the conditions L(c;a) = L(d;a) = 1 is 


(e4 re(a) _ @¢ r2(a)) eters (a) +4 (ec T1(a) _ ¢d Fata) etor2(a) 
L(xo; a) = ef Tila)td rela) — ec r2(a)td ry (a) 


(4.234) 


force <2 <d. 
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Inverting the Laplace transform above is not easy. However, in this type 
of problem, we must often content ourselves with explicitly calculating the 
moment-generating function of the first-passage time considered. 


Suppose, to simplify, that there is a single boundary, at d (which is tanta- 
mount to taking c = —oo) and that pz = 0. Since, from Eq. (4.232), L(ro; a) 
must be in the interval [0,1], for any zp < d, we must discard the solution 
with r1(q@) in (4.233). Indeed, with py = 0, this solution is 


~V2axr0/o 


cyje — Cy X CO as To — -C 


We must therefore choose the constant c; = 0. We then have 


L(20;) = cgeV?0%0/¢ 


and the condition L(d; a) = 1 implies that 


L(xo; a) = eV20(%-9/2 Yay <d (4.235) 


Remark. We obtain the same result by taking the limit as c decreases to —oo 
in the formula (4.234) (with us = 0). 


We can check that the inverse Laplace transform of the function L in Eq. 
(4.235) is given by (see the inverse Gaussian distribution, p. 216) 
(d — xo) (d — 0)? 
tij= 
frst) = Taegaes P| aa 


The density function of the random variable Ty in the general case where 
ME Ris 


} for t > 0 


a, tipi, fe 2 
fri(t) = a ex { id a ut) \ for t > 0 (4.236) 


and the function L becomes 


L(xo;@) = exp {Sore lH — (p+ 2a07)'/?] \ forz%o <d (4.237) 


After having obtained the moment-generating function of the random vari- 
able Tg, we can calculate the probability that this variable is finite, as follows: 


Pilg = 00) = a L (29; a) (4.238) 


Example 4.4.2. In the preceding example, with a single boundary and p = 0, 
we obtain that 
P(Ta < oo] = une 2a(ro—d)/o i 1 
a 
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which generalizes (in the case when d > 0) the result already mentioned 
regarding the standard Brownian motion. 
However, when pp # 0, we deduce from the formula (4.237) that 


_ yk (d — £0) 2 2\1/2 
Pit <0) limexp { “Fe? [e- (uu + 2a0~) 


= 1 if >O0 
~ | e2u(d—-20)/07 ig u<O 


To obtain the moments of the random variable Tz, we can use the formula 
(see p. 19) 


10) n o” 
ET?) = (-1) aan L(x; )|,=0 (4.239) 
for n = 1,2,.... Thus, in Example 4.4.1, if c = —oo and p > 0, we calculate 


[see the formula (4.237)] 


O d— 
E [Taj = ~ Fe exP {<2 lu — (uw? + 2a0%)'/2| | : 
= —L(z9; a) (* <3) (-1)(u? + 2a07)~ 70? F 
d— Zo 
= 4,240 
a ( ) 


Remark. We can also try to solve the ordinary differential equation satisfied 
by the function 


Mn,a(to) := E[TZ] for n = 1,2,... (4.241) 
namely, 
1 (0) S prrinsalna) + tla) mina (Bo) = (xo) (4.242) 
Q” Zo dae n,d\ Zo m{x0) a Mind To) = —N Mn—-1,d\Z0 : 


under the boundary condition m,,a(d) = 0. In particular, we have 


1 d? d 
3 V0) Gagm.a(zo) + m(z0) Fm1,a(to) =-l (4.243) 


When there are two boundaries, if we wish to calculate the probability 
that the process will hit the boundary at d before that at c, we can simply 
solve the ordinary differential equation 


2 


1 d d 
3°(70) GagPalzo) + ™m(xo) Galo) = 0 (4.244) 


4.4 First-passage problems 221 
where 
pa(®o) = P[X(Te,a(x0)) = d} (4.245) 


This differential equation is obtained from Eq. (4.230) by setting a = 0. 
Indeed, we may write that 


p(t; 20) = pe(t; x0) + palt; xo) (4.246) 


where p,(t; xo) (respectively, pa(t;xo)) is the density function of the random 
variable T; (resp., T’,) denoting the time the process takes to go from x to c 
(resp., d) without hitting d (resp., c). Let 


La(xo3 a) = i e % pa(t; x0) dt (4.247) 
0 
We have 
P[X(Te,a(x0)) = d] = P[T;(z0) < oo] = is pa(t;2o) dt = La(xo;0) (4.248) 


Now, the function La(zo; a) also satisfies Eq. (4.230). 
The differential equation (4.244) is valid for c < ro < d. The boundary 
conditions are 


pa(d)=1 and pa(c) =90 (4.249) 


Example 4.4.3. If {W(t),t > 0} is a Wiener process with diffusion coefficient 
a”, then we must solve 
a? d? 
eesti, =0 
2 dexgha(o) 
We find at once that 
Pa(Lo) = 1X0 + Co 


where co and c; are constants. The boundary conditions imply that 


th, = 
Ppa(Zo) = 7 forc<2%<d 
Note that this formula does not depend on o?. Moreover, if c = —d, then we 


have that pg(0) = 1/2, which could have been predicted, by symmetry. 


Until now, we only considered the time a diffusion process takes to reach 
a boundary or either of two boundaries. We can define these boundaries as 
being absorbing. If the boundary at c is reflecting, it can be shown that the 
boundary condition becomes 
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0 
T(x: =0 4.250 
Axo (£0; a) ( ) 


Toc 


Finally, we can also try to calculate first-passage time distributions in two 
or more dimensions. However, the problem of solving the appropriate partial 
differential equation is generally very difficult. 


Example 4.4.4. If {B(t),t > 0} is a standard Brownian motion, starting from 
bo, and if {X(t),t > 0} is its integral, we find that the Kolmogorov backward 
equation satisfied by the function 


P(Ty (bo, € (t,t + dt 
p(t; bo, Zo) := PiFalborto) st + al 


where 
Ta(bo, 20) := min{t > 0: X(t) =d >0| B(0) = bo, X(0) = xo < d} 
is the following: 
12%, Oe _ a 
202" "Ax Ot 
The main difficulty is due to the fact that the process {X (t),t > 0} cannot hit 


the boundary X(t) = d (for the first time) with B(t) < 0. Indeed, we deduce 
from the formula 


x()=20+ [BL as = £ X(t) = Blt) 


that if B(t) takes on a negative value, then X(t) decreases. Consequently, 
the process {X(t),t > 0} can attain a value d (> 0) greater than zo at time 
Ta(bo, Zo) only if B[Ta(bo, r9)} > 0, so that the function p is not continuous 
on the boundary. 
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Section 4.1 


Remark. In the following exercises, the process {B(t),t > 0} is always a stan- 
dard Brownian motion. 


Question no. 1 
We define 


X(t) = B’(t) fort >0 


(a) Is the stochastic process {X(t),t > 0} a Wiener process? Justify. 
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(b) Is {X(¢),¢ > 0} (i) wide-sense stationary? (ii) mean ergodic? Justify. 


Question no. 2 
We consider the stochastic process {X(t),¢ > 0} defined by 


X(t) = B(t)+ B(t?) fort >0 


(a) Calculate the mean of X(t). 

(b) Calculate Cov[X(t), X(t + 7)], for 7 > 0. 

(c) Is the stochastic process {X(t),t > 0} (i) Gaussian? (ii) stationary? (iii) a 
Brownian motion? Justify. 

(d) Calculate the correlation coefficient of B(t) and B(t?), for t > 0. 
Question no. 3 


Calculate the variance of the random variable X := B(t) — 2B(r), for 
O<t<r. 


Question no. 4 
Let X(t) := |B(t)|, for t > 0. Is the stochastic process {X(t),t > 0} 
(a) Gaussian? (b) stationary? Justify. 


Question no. 5 
Let {X(t),¢ > 0} be the stochastic process defined by 


X(t) = B(t+1)-B(1) fort >0 


(a) Calculate the autocovariance function of the process {X(t),t > O}. 


(b) Is the process {X(t),¢ > 0} (i) Gaussian? (ii) a standard Brownian mo- 
tion? (iii) stationary? (iv) mean ergodic? Justify. 


Question no. 6 
Calculate the variance of X := B(4) — 2B(1). 


Question no. 7 
Is the stochastic process {X(t),t > 0} defined by 


X(t) =—-—B(t) fort >0 


Gaussian? Is it a Brownian motion? Justify. 


Question no. 8 
Let {X(t),t > 0} be a Gaussian process such that X(0) = 0, B[X(t)] = ut 
if ¢ > 0, where yp 4 0, and 


Rx(t,t +7) =2t+p?t(t+7) fort,r >0 


Is the stochastic process {Y(t),t > 0}, where Y(t) := X(t) — pt, a Brownian 
motion? Justify. 
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Question no. 9 
Is the Wiener process mean ergodic? Justify. 


Question no. 10 
We set 


Y(t) = +B) for t > 0 


(a) Is the process {Y(¢),t > 0} Gaussian? Justify. 
(b) Calculate the mean of Y(t). 

(c) Calculate Cov[Y(s), Y(t)], for0<s <t. 
Indication. If X ~ N(0,0%) and Y ~ N(0,o%), then 


E[X°?Y?] = E[X?|E[Y?] + 2(E[XY])” 


(d) Is {Y(t),¢ > 0} a wide-sense stationary process? Justify. 


Question no. 11 
Consider the stochastic process {X(t),t > 0} defined by 


X(t) = Bit) -— Bd) 


where {t] denotes the integer part of f. 
(a) Calculate the mean of X (é). 


(b) Calculate Cov[X (t:), X(t2)], for tg > t,. Are the random variables X(t1) 


and X(t2) independent? Justify. 


(c) Is the stochastic process {X(t),t > 0} (i) stationary? (ii) a Brownian 


motion? Justify. 


Question no. 12 


At each time unit, the standard Brownian motion {B(t),t > 0} is shifted 
to {B(n)|, where { ] denotes the integer part. Let X, be the position at time 
n, forn =0,1,2,.... Then the process {X,,n = 0,1,...} is a (discrete-time) 
Markov chain. Calculate (a) p;,;, for 1,7 > 0 and (b) P[X, = 0, X2 = 0]. 


Question no. 13 
We define 
X(t) = Biln(t+1)) fort >0 


(a) Is the stochastic process {X(t),t > 0} a Brownian motion? Justify. 


(b) Calculate E[X?(t) | X(t) > 0]. 


Question no. 14 
Let {X(t),t > 0} be the stochastic process defined by 


x) = Et 20 Vt>0 
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where € is a positive constant. 

(a) Calculate Cy (t,t +s), for s,t > 0. 

(b) Is the process {X(t),t > 0} (i) Gaussian? (ii) stationary? (iii) a Brownian 
motion? (iv) mean ergodic? Justify. 

Question no. 15 


Let X1, X2, ... be independent and identically distributed random vari- 
ables such that px, (x) = 1/2 if « = —1 or 1. We define Yo = 0 and 


n 
Vee. Xe. tora 2. 
k=1 


Then the stochastic process {Y;,,n = 0,1,...} is a Markov chain (see p. 85). 
We propose to use a standard Brownian motion, {B(t),t > 0}, to approx- 

imate the stochastic process {Y,,n = 0,1,...}. Compare the exact value of 

P[¥30 = 0] to P[-1 < B(30) < 1]. 

Indication. We have that P|N(0, 1) < 0.18] ~ 0.5714. 


Section 4.2 


Question no. 16 
Let {U(t),¢ > 0} be an Ornstein—Uhlenbeck process defined by 


Ups e Re") 


(a) What is the distribution of U(1) + U(2)? 
(b) We set 


(i) Calculate the mean and the variance of V(t). 
(ii) Is the process {V(t),¢ > 0} Gaussian? Justify. 


Question no. 17 

Suppose that {X(t),t > 0} is a Wiener process with drift coefficient M and 
diffusion coefficient ¢? = 1, where M is a random variable having a uniform 
distribution on the interval 0, 1]. 


(a) Calculate E[X(t)] and Cov[X(s), X(t)], for s,t > 0. 
(b) Is the process wide-sense stationary? Justify. 
Question no. 18 

Let {Y(t),t > 0} be a geometric Brownian motion. 


(a) Show that the density function of the random variable Y(¢) is given by 
the formula (4.52). 


(b) Is the process {Y(¢),t > 0} stationary? Justify. 
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Question no. 19 


Let {X(t),t > 0} be a Wiener process with drift coefficient p > 0 and 
diffusion coefficient o? = 1. 


(a) (i) Calculate, as explicitly as possible, E[X(t) | X(t) > 0] in terms of 
Q(x) := P[N(0,1) > a]. 

(ii) Calculate E[X(t)X (t+ s)], for s,t > 0. 
(b) Let Z7(0) = 0 and 


Oye fort >0 
pt 


Is the stochastic process {Z(t),t > 0} a Brownian motion? Justify. 


Section 4.3 


Question no. 20 
(a) Calculate the covariance of X and Y, where 


1 1 
X = | tdB(t) and Y = f t? dB(t) 
—1 -1 


(b) Are the random variables X and Y independent? Justify. 


Question no. 21 
We define the stochastic process {Y(t),t > 0} by 


Y(t) = [ dB(s) fort >0 


(a) Calculate the autocovariance function of the process {Y(t),t > 0}. 

(b) Is the process {Y(t),t > 0} Gaussian? Justify. 

(c) Let {Z(t),t > 0} be the stochastic process defined by Z(t) = Y?(t). 
Calculate its mean. 


Question no. 22 


Calculate the mean and the autocovariance function of the stochastic pro- 
cess {Y(t),t > 0} defined by 


t 
Y(t) = oy e* dW(s) fort >0 
0 


where c is a real constant. 
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Section 4.4 


Question no. 23 
Let {W(t),t > 0} be a Brownian motion with infinitesimal parameters 
p= 0 and a? (> 0). We set 


Y(t)=W(t?) fort >0 


(a) Is the stochastic process {Y(t),t > 0} a Brownian motion? Justify. 
(b) Calculate Cov[Y(s), Y(t)], forO<s <t. 
(c) Let Ty := min{t > 0: Y(t) = d > 0}. Calculate the probability density 
* function of the random variable Ty. 
Question no. 24 

Let {Y(t),t > 0} be the stochastic process defined in Question no. 21. 
(a) Calculate the distribution of Y(1) + Y(2). 
(b) Let ZT) := min{t > 0: Y(t) = 1}. Calculate the probability density function 
of the random variable T). 


Question no. 25 
We consider the process {X (t),t > 1} defined by 


X(t) =e"/*B(el/*) fort >1 


Suppose that B(e) = 0, so that X(1) = 0. 
(a) Calculate ELX(t)] and Cov[X(t), X(t + s)], fort > 1,8 > 0. 


(b) Is the stochastic process {X(t),t > 1} (i) Gaussian? (ii) an Ornstein— 
Uhlenbeck process? (iii) stationary? Justify. 


(c) Let Ty := min{t > 1: X(t) =d > 0}. Calculate fr,(t). 


Question no. 26 
Let {Z(t),0 <t <1} be a Brownian bridge. We define 


t 
y= | Z(r) dr forO<t<1 
0 


(a) Calculate E[Y(t)] and Cov[Y(t), Y(t+s)], forO<t<1,s2>0,s+t<1. 
(b) Is the stochastic process {Y(t),0 < t < 1} (i) Gaussian? (ii) stationary? 
(iii) a Brownian bridge? Justify. 

(c) Calculate approximately, if d > 0 is small, the probability that the process 
{Y(t),0 <¢ <1} will reach d in the interval (0,1). 


Question no. 27 
Let T be the first-passage time to the origin for a standard Brownian 
motion starting from c > 0. We define S = 1/Ty. 
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(a) Calculate the probability density function of S. What is the distribution 
of the random variable S? 


(b) Find a number b (in terms of the constant c) for which we have P[S < }| 
= 0.5. 


Question no. 28 
The nonstationary Ornstein-Uhlenbeck process {X(t),t > 0} is a Gaus- 
sian process such that, if 7 = 1 (see Subsection 4.2.5), 


Th en 2at 
E[X(t)] = X(0)e"® and V[X(t)] = 5a 


where a > 0 is a parameter. Suppose that X(0) = d > 0. We define Ty(d) = 
min{t > 0: X(t) = 0}. Calculate fr,(a)(t) when a = 1/2. 


Question no. 29 

Let {W(t),t > 0} be a Brownian motion with drift coefficient uz and diffu- 
sion coefficient o?. We assume that the flow of a certain river can be modeled 
by the process {X(t),t > 0} defined by 


Xit)h=e"O- Vt S0 


where & is a constant. Next, let d be a value of the flow above which the risk 
of flooding is high. Suppose that X(0) = d/3. Calculate the probability that 
the flow will reach the critical value d in the interval (0,1] if uz > 0 and o =1. 


Question no. 30 
Let {X(t),t > 0} be an Ornstein-Uhlenbeck process for which a = 1 and 
o* = 2, and let {Y(t),t > 0} be the process defined by 


t 
Y(t) =f X(s) ds 
0 
Finally, we set 
Z(t) = X(t)+Y(t) fort >0 


(a) Calculate E[Z(t)]. 
(b) Calculate Cov[Z(t), Z(t + s)], for s,t > 0. 
Indication. We find that Cov[X(t), X(t + s)] =e7* and 


Cov[Y(t), Y(t + s)] = 2t-1+e7*—e~* +e" 849 for s,t >0 
(c) Is the stochastic process {Z(t),t > 0} stationary? Justify. 


(d) Let Ty(z) := min{t > 0: Z(t) = d | Z(0) = z (< d)}. Calculate the 
probability density function of Ty(z). 
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Question no. 31 


Let {X(t),¢ > 0} be a Brownian motion with drift coefficient js and diffu- 
sion coefficient o. 


(a) Suppose that » = 0 and that o? is actually a random variable V having a 
uniform distribution on the interval (0,1). 

(i) Calculate ELX(t)] and Cov[X(s), X(¢)], for s,¢ > 0. 

(ii) Is the stochastic process {X(t),t > 0} Gaussian and stationary? Jus- 
tify. 


(b) Suppose now that ps > 0 and g? = 1. Let re 1(Zo) be the time the process 
takes to attain 1 or —1, starting from zo € [—1, 1]. 

(i) It can be shown that m_1,1(z0) := BIT, 1(o)] satisfies the ordinary 
differential equation 


1d? d 
get (20) oe Ham -1,1(a0) =-1 

0 
Solve this differential equation, subject to the appropriate boundary condi- 
tions, to obtain m_1,1(%o) explicitly. 

(ii) Similarly, the function p_1.1(ao) := P[X(T_1,1(zo0)) = 1] is a solution 
of 

1d d 

2 daz 11(@0) + Ha p-1,1(0) = 0 


Obtain an explicit formula for p_1,1(%o). 
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Poisson Processes 


5.1 The Poisson process 


We already mentioned the Poisson process in Chapters 2 and 3. It is a par- 
ticular continuous-time Markov chain. In Chapter 4, we asserted that the 
Wiener process and the Poisson process are the two most important stochas- 
tic processes for applications. The Poisson process is notably used in the basic 
queueing models. 

The Poisson process, which will be denoted by {N(t),t > O}, is also a pure 
birth process (see Subsection 3.3.4). That is, N(t) designates the number of 
births (or of events, in general) that occurred from 0 up to time t. A process 
of this type is called a counting process. 


Definition 5.1.1. Let N(t) be the number of events that occurred in the inter- 
val [0,t]. The stochastic process {N(t),t > 0} is called a counting process. 


Counting processes have the following properties, which are deduced di- 
rectly from their definition. 


Properties. i) N(t) is a random variable whose possible values are 0,1,.... 
ii) The function N(t) is nondecreasing: N(t2) — N(t1) > 0 if tg > t, > 0. 
Moreover, N(t2) — N(t,) is the number of events that occurred in the interval 
(t1, ta]. 


Definition 5.1.2. A Poisson process with rate X (> 0) is a counting pro- 
cess {N(t),t > 0} having independent increments (see p. 50), for which 
N(0) =0 and 


N(r +t) — N(r) ~ Poi(At) V7,t>0 (5.1) 
Remarks. We deduce from the preceding formula that a Poisson process also 


has stationary increments (see p. 50), because the distribution of N(7 + t) — 
N(r) does not depend on 7. Moreover, by taking 7 = 0, we may write that 
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N(t) = N(04+t) — N(O) ~ Poi(At) Vi>O (5.2) 


ii) If we specify in the definition that {N(t),¢ > 0} is a process with indepen- 
dent and. stationary increments, then we may replace the formula (5.1) by the 
following conditions: 


P[N(5) = 1] = 6 + 0(6) (5.3) 
P[N(5) = 0] =1—26 + 0(6) (5.4) 


where o(4) is such that (see p. 125) 


. 0(6) 
—-+= 5.5 
he aaa O29) 
Thus, the probability that there will be exactly one event in an interval of 
length 6 must be proportional to the length of the interval plus a term that is 
negligible if 6 is sufficiently small. Furthermore, we have 


P[N(6) > 2] =1-{P[N(6) =0] + PIN(5) = 1} =0(5) (5.6) 


Therefore, it is not impossible that there will be two or more events between 
an arbitrary to and typ + 6. However, it is very unlikely when 6 is small. 


It is not difficult to show that if N(5) ~ Poi(Ad), then the conditions (5.3) 
and (5.4) are satisfied. We have 
(46)? 


P[Poi(Ad) = 0] = e~*® = 1—AS + Sophie 6 +0(6) (5.7) 


and 
P[Poi(Ad) = 1] = Ade~*? = AS[1 — Ad + 0(5)] = Ad + 0(6) (5.8) 


As will be seen in Section 5.2, in the more general case where A = A(¢), it can 
also be shown that if the conditions (5.3) and (5.4) are satisfied (and if the 
increments of {N(t),t > 0} are stationary), then the formula (5.1) is valid. 
Consequently, we have two ways of determining whether a given stochastic 
process is a Poisson process. 


Since the random variable N(t) has a Poisson distribution with parameter 
At, for all t > 0, we have 


E|N(t)] = At (5.9) 
E[N?(t)| = V[N(t)] + (E[N(2)]})? = At + 70? (5.10) 


As we did in the case of the Wiener process, we use the fact that the incre- 
ments of the Poisson process are independent (and stationary) to calculate its 
autocorrelation function. We may write, with the help of the formula (5.10), 
that 
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Rn (t,t + s):=E[N(t)N(t + s)] = E[N(t){N(t +s) — N(t)}] + EIN? (8)] 


"SE(N(t)|E[N(t + s) — N(t)] + E[N?(t)] 
= Mths + (At + A7t?) = A7e(t + 8) HAE (5.11) 
It follows that 
Cy (t,t +s) = Rn(t,t +s) —At{A(t+s)] =At (5.12) 


For arbitrary values of t; and tg, the autocovariance function of the Poisson 
process with rate A > 0 is given by 


Cn (ty, ta) = Amin{t, te} (5.13) 


Remarks. i) Note that the formula above is similar to that obtained for the 
Wiener process (see p. 178). Actually, the stochastic process {N*(t),t > 0} 
defined by 


N*(t)= N(t)—At fort >0 (5.14) 


has zero mean and an autocovariance function identical to that of a Wiener 
process with diffusion coefficient ¢? = \. We can also assert that the Poisson 
process, {N(t),t > 0}, and the Brownian motion with drift, {X(t),t > 0}, 
have the same mean and the same autocovariance function if wy = 0? = X. 
Moreover, by the central limit theorem, we may write that 


Poi(At) © N(At, At) (5.15) 


if At is sufficiently large. 


ii) Since the mean of the Poisson process depends on the variable t, this process 
is not even wide-sense stationary (see p. 53), even though its increments are 
stationary. Furthermore, its autocovariance function depends on t; and on fa, 
and not only on |t2 — t,|. 


Example 5.1.1. Suppose that the failures of a certain machine occur according 
to a Poisson process with rate \ = 2 per week and that exactly two failures 
occurred in the interval [0,1]. Let to (> 3) be an arbitrary value of t. 

(a) What is the probability that, at time to, (at least) two weeks have elapsed 
since (i) the last failure occurred? (ii) the penultimate failure occurred? 

(b) What is the probability that there will be no failures during the two days 
beginning with to if exactly one failure occurred (in all) over the last two 
weeks? 


Solution. In order to solve a problem on the Poisson process, we must first 
determine the value of the parameter of the Poisson distribuéson for each 
question asked. Let N(t) be the number of failures in the interval [0, t], where 
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t is measured in weeks. Since the average arrival rate of the failures is equal 
to two per week, we have 
N(t) ~ Poi(2t) 


(a) (i) In this question, we are interested in the number of failures during a 
two-week interval. As the Poisson process has stationary increments, we can 
calculate the probability asked for by considering the random variable N(2) ~ 
Poi(4). We seek 


PIN (2) = 0] = e7* = 0.0183 (5.16) 


(ii) Two weeks or more have elapsed since the penultimate failure if and 
only if the number of failures in the interval {to — 2, to] is either 0 or 1. There- 
fore, we seek 

P[N(2) < 1] = e~4(14+ 4) ~ 0.0916 


(b) Since the Poisson process has independent increments, the fact that there 
has been exactly one failure over the last two weeks does not matter. We are 
interested in the value of 


N (to + 2/7) — N(to) & N(2/7) ~ Poi(4/7) 
We calculate 
P{N(2/7) = 0] = e~*/" ~ 0.5647 
Remarks. i) We could have written instead that A = 2/7 per day, and then, 
in (b), we would have had that N(2) ~ Poi(4/7). 


ii) In this problem, we assume that the rate of the Poisson process is the 
same for every day of the week. In practice, this rate is probably different on 
Sundays than on Mondays, for instance. It also probably varies at night from 
its value during the day. If we want to make the problem more realistic, we 
must use a parameter \ that is not a constant, but rather a function of f, 
which will be done in Section 5.2. 


Proposition 5.1.1. Let {Nj (t),t > 0} and {No(t),t > 0} be two independent 
Poisson processes, with rates \1 and 2, respectively. The process {N(t),t > 0} 
defined by 


N(t) = Ni(t)+ No(t) Vt>0 (5.17) 
is a Poisson process with rate  := A, + Ag. 
Proof. First, we have 
N(0) := Ni(0) + No(0) =04+0=0 (5.18) 


as required. 
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{No(t),t > 0} are independent, we can check that so are those of the pro- 
cess {N(t),t > 0}. We simply have to write that 


Next, given that the increments of the processes {Ni(t),t > O} and 


N(ty) — N(te—1) = [Ni(te) — Ni(te-1)] + [Na(te) — No(te-1)] (5-19) 


Finally, the sum of independent Poisson random variables, with parameters 
Q@, and ap, also has a Poisson distribution, with parameter a := a, + Qo. 
Indeed, if X; ~ Poi(a;), then its moment-generating function is given by (see 
Ex. 1.2.8) 


Mx, (t) = e~% exp {e*a;} (5.20) 
It follows that 
Mx, +x2(t) '"S Mx, (t)Mx, (t) = e7(*%) exp {e' (ay + a2) } (5.21) 
Thus, we may write that 


N(r +t) — N(r) = [Ni(r +t) — Ni(r)] + [No(r + t) — No(r)] 
~ Poi ((A1 +A2)t) Vr7,t>0 O (5.22) 


The preceding proposition can, of course, be generalized to the case when 
we add j independent Poisson processes, for j = 2,3,... . Conversely, we can 
decompose a Poisson process {N(t),t > 0} with rate into j independent 
Poisson processes with rates \; (i = 1,2,...,j), where Ay +...+ A; =A, 
as follows: suppose that each event that occurs is classified, independently 
from the other events, of type i with probability p;, for i = 1,...,7, where 
pi +...+p; = 1. Let N;(t) be the number of type i events in the interval 
[0, t]. We have the following proposition. 


Proposition 5.1.2. The stochastic processes a (t),¢ > O} oie above are 
independent Poisson processes, with rates Aj := a SOS = 9k 


Proof. Since N(t) = S~/_, Ni(t), we may write that 


P[Ni(t) =MN1,... Nj (t) == a 


= 5° P[Ni(t) = m,... Nj (t) = nj | N(t) =k] PIN(t) = ki 
k=0 

= P[N\(t)=m1,...,Nj(t) =n; | N(t) =m +..-+ 75] 
x P[N(t) =n t+... +n,| (5.23) 


Let n:=1n,+...+2,;. The conditional probability above is given by 


P(Na(t) = m1... .Ng(t) =n | N(t) =n] = ealua (5.24) 
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This is an application of the multinomial distribution (see p. 4), which gener- 
alizes the binomial distribution. It follows that 


(At)” 
oot 
PIM =m OHI (Se salle ye ae 
j ayn 
2s [Teor Qe" (5.25) 
i=1 
from which we find that 
P{N,(t) = ni] = ew rvee pty fori=1,...,9 (5.26) 
4° 
and 
j 
P(Ni(t) = m,... ,Nj(t) = nj] = [[ PINE) = 24] (5.27) 


Therefore, we can assert that the random variables N;(t) have Poisson distri- 
butions with parameters Ap,t, for i=1,... ,j, and are independent. 

Now, given that {N(t),t > 0} has independent and stationary increments, 
the processes {N;(t),t > 0} have independent and stationary increments V i 
as well. It follows that N,;(7 + t) — Ni(r) ~ Poi(Ap,t), for? =1,... ,7. 

Finally, since N(0) = 0, we have that N,(0) = ... = N;(0) = 0. Hence, 
we may conclude that the processes {N;(t),t > 0} satisfy all the conditions 
in Definition 5.1.2 and are independent. 0 


Remarks. i) We can use the other way of characterizing a Poisson process, 
namely the one when we calculate the probability of the number of events in 
an interval of length 6. We have 


P[N,(5) = 1] = P[Ni(6) = 1] N(4) = 1] P[N(S) = J] 
+P[N;(5) = 1] N(6) > 1] P[N(6) > 1) 
= pied + 0(5) = pi[Ad + 0(6 )} + o(6) 

= Ap;d + o(d) 


Moreover, we may write that 
P[N;(5) = 0] = 1 — Apid + 0(4) (5.28) 
because N;(d) < N(6), for all ¢, which implies that 
P[Ni(8) = 2] < PIN(8) > 2 = 016) (5.29) 


We could complete the proof and show that the processes {N;(t),¢ > 0} are 
indeed independent Poisson processes, with rates Ap;, for 7 = 1,... ,7. 
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Fig. 5.1. Example of a trajectory of a Poisson process. 


ii) The fact than an arbitrary event is classified of type i must not depend on 
the time at which this event occurred. We will return to this case further on 
in this section. 


Since a Poisson process is a particular continuous-time Markov chain, we 
can assert that the time 7; that the process spends in state i € {0,1,...} 
has an exponential distribution with parameter vy; > 0 (see p. 121) and that 
the random variables 79, 71, ... are independent (by the Markov property). 
Furthermore, because the Poisson process has independent and stationary in- 
crements, the process starts anew, from a probabilistic point of view, from 
any time instant. It follows that the 7;’s, for 7 = 0,1,..., are identically dis- 
tributed. All that remains to do is thus to find the common parameter of the 
random variables 7;. We have 


Plro > t] = P[N(t) = 0] = P[Poi(At) = 0] = e~** (5.30) 


= f(t) = “ (1—e"*) =e? fort > 0 (5.31) 


That is, 7 ~ Exp(A). Therefore, we can state the following proposition. 


Proposition 5.1.3. Let {N(t),t > 0} be a Poisson process with rate X, and 
let 7% be the time that the process spends in state i, fori = 0,1,.... The 
random variables 7), 71, ... are independent and 7; has an exponential Exp(X) 
distribution, for all i. 


Notation. We designate by T;, To, ... the arrival times of the events of the 
Poisson process {N(t),¢ > 0} (see Fig. 5.1). 


Corollary 5.1.1. In a Poisson process with rate X, the time needed to obtain 
a total of n events, from any time instant, has a gamma distribution with 
parameters a =n and xX. 
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Proof. The arrival time T;,, of the nth event, from the initial time 0, can be 
represented as follows (because {N(t),t > 0} is a continuous-time stochastic 
process): 


n-l 
T= oie fernat2 kk (5.32) 

i=0 
Since the 7,’s are independent random variables, all of which having an ex- 
ponential distribution with parameter \, we can indeed assert that T, ~ 


G(a@ = n,.) (see Prop. 3.3.6). Moreover, we deduce from the fact that the 
Poisson process has independent and stationary increments that 


PlTr+k <tt+ty | N(tx) = k] a PIT, < t] for all k =0,1,... (5.33) 


That is, if we know that at a given time t, there have been exactly k events 
since the initial time, then the time needed for n additional events to occur, 
from t,, has the same distribution as T,. 0 


The preceding results and the following relation: 
N(it)>n — Thkxt (5.34) 


provide us with yet another way of defining a Poisson process. This alternative 
definition may be easier to check in some cases. 


Proposition 5.1.4. Let X; ~ Exp(d), fori = 1,2,..., be independent ran- 
dom variables, and let Ty := 0 and 


n 
Ty i=) Kye for WS 12s: (5.35) 
i=l 
We set 
N(t) = max{n > 0: T, < t} (5.36) 


Then, {N(t),t > 0} is a Poisson process with rate X. 
Remarks. i) Since P{T, > 0] = 1, we indeed have 
N(0) = max{n > 0:T, <0} =0 (5.37) 
ii) We also have 
N(th=n => {T, <t}N {Tri > t} (5.38) 


from which we obtain the definition of N(t) in terms of T,, above. 


iii) To calculate the probability of the event {N(t) = n}, it suffices to notice 
that 
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P[N(t) =n] = P[N(t) > n] — P[N(t) >n4+1) = PlTh <t] — PiTn41 < tl 
(5.39) 


and to use the generalization of the formula (3.173) to the case when Y has 
a G(n, A) distribution: 


PIY <yJ=1-P[W<n-1])=P[W>n], where W ~ Poi(Ay) (5.40) 
We indeed obtain that 


PIN(t) =n] = P[Tr < t]— P[Tr41 < t] (5.41) 
= P[Poi(At) > n] — P[Poi(At) > n + 1] = P[Poi(At) = n] 


Tx := min{t > 0: N(t) > d (> 0)} (5.42) 


If d € {1,2,...}, then we deduce from the relation (5.34) that Tj ~ G(d, A). 
When d ¢ {1,2,...}, we only have to replace d by [d] +1, that is, the smallest 
integer larger than d. Note that, since limzy_,.. N(t) = oo, we may write that 
P(T} < oo] = 1, for any real number d € (0, 00). 

A more difficult problem consists in finding the distribution of the random 
variable 


Toad := min{t > 0: N(t) > ct +d} (5.43) 
where c >O andd>0. 


Example 5.1.2. Suppose that the random variables X; have a uniform distri- 
bution on the interval (0,1], rather than an exponential distribution, in the 
preceding proposition. To obtain a Poisson process by proceeding as above, it 
suffices to define 


1 
Y= —z xX fori =1,2,... 


Indeed, we then have, for y > 0: 
PIY; Sy] = Pin X, > -ry) = P[X, >] =1-e 
so that 
fy(y) =A for y = 0 


Thus, the stochastic process {N(t),t > 0} defined by N(t) = 0, fort <Y%, 
and 


N(t) = max {n> 1: ov <e} fort >Y (5.44) 


i=1 


is a Poisson process with rate A. 
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Remark. If we define Yo = 0, then we may write that 


N(t) =max {n> 0: ov <¢} Vt>0 (5.45) 
i=0 


Proposition 5.1.5. Let {N(t),t > 0} be a Poisson process with rate ’. We 
have that T; | {N(t) = 1} ~ U(0,t], where T, is the arrival time of the first 
event of the process. 


Proof. For 0 < s < t, we have 


Pit, <s|(t)—1) = PM S8NO=1 


PIN(t) = 1] 
P(N(s) = 1, N(t) — N(s) = 0] 
PIN(@) =] 
ind. (Ase 8) e—Alt-8) 8 
Ate7*t t 


Remarks. i) The result actually follows from the fact that the increments of 
the Poisson process are independent and stationary. This indeed implies that 
the probability that an event occurs in an arbitrary interval must depend only 
on the length of this interval. 


ii) More generally, if T* denotes the arrival time of the only event in the 
interval (t,,t2], where 0 < t; < te, then T* is uniformly distributed on this 
interval. 


iii) The random variable T; | {N(t) = 1} is different from Ty := T; | {T; < t}. 
The variable Ty has a truncated exponential distribution: 
P(t, <s] 1-e7*” 


Pitt so=PIssisd= pe sy = [oee forO<s<t 
i a 
(5.46) 


Note that the occurrence of the event {N(t) = 1} implies the occurrence of 
{T, < t}. However, {T; < t} => {N(t) > 1}. On the other hand, we may 
write that 


T, | {N(t)=1} =T, | {T, <t,To > t} (5.47) 
We would like to generalize the preceding proposition by calculating the 


distribution of (T,... ,T;,,), given that exactly n events occurred in the in- 
terval (0, t]. Let us first consider the case when n = 2. Let 0 < ti < te <t. 
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To obtain the conditional distribution function of the random vector (T;,T>), 
given that N(t) = 2, we begin by calculating 


PIT, < ti, T2 < to, N(t) = 2] 


= P[N(ti) = 1, N(tz) — N(t1) = 1, N(t) — N(t2) = 0] 
+P[N(t1) = 2,.N(t) — N(t1) = 0] 


2 
= (tie At, = je Meter NG=8)) 4 Cy 


t eMigntet)) 
2 
= (alt _ t1) + ar) ert 


(5.48) 


where we used the fact that the increments of the process {N(t),t > 0} are 
independent and stationary, from which we find that 


PIT, < t1,T2 < tz | N(t) = 2] 


_ PIT, < ti, To < te, N(t) = 2| 


- P{N(t) = 2] 
_ (Ae (ta — 1) + 3 (At1)?) eo 
a $(At)2e- 
7 2t) (te —t)) +t? 


3 (5.49) 
so that 
(oe 2t1(t2 — t)) t? 
Fr,,72|N(t) (tr, te | 2) = Dt ate age + 2 
2 2 
=gt0=% for0 <ty <te<t (5.50) 


Remark. From the preceding formula, we calculate 


t 
2 2(¢—t 
Fryjnce) (4 | 2) =/ p dtz = eu for0<t, <t (5.51) 
1 


Note that the distribution of T; | {N(t) = 2} is not uniform on the interval 
(0, t], contrary to that of T; | {N(t) = 1}. We also have 


te 
2 at 
Frain(e (t2 | 2) =| p d= = 
0 


2 forO0<to<t 


(5.52) 
Finally, we may write that 


Sr, 79| w(t) (1, ta | 2) 
freiT,,N(t) (te | t1,2) = ax) = 


2/t? 
fryney(t1 | 2) 


2(t — t1)/t? 
for0<t)<te<t 


(5.53) 
— eb] 
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That is, Tz | {T, = t1,N(t) = 2} ~ U(t1,¢#], from which we deduce that 
(T2 = T,) {T, = t1, N(t) — 2} om U(0,t — ty]. 


To obtain the formula in the general case, we can consider all the possible 
values of the random variables N(ti), N(t2) — N(t1), .-., N(tn) — N(tn-1), 
given that T; < t,,T2 < te,..., Ty <tn, N(t) =n. The total number of ways 
to place the n events in the intervals (0,¢,], (ti, ta], ---, (tr—-1,tn] is given 
by the multinomial coefficients (see p. 4). Moreover, N(t,;) must be greater 
than or equal to 1, N(¢2) must be greater than or equal to 2, etc. However, 
it suffices to notice that the only nonzero term, after having differentiated 
the conditional distribution function with respect to each of the variables 
t,,...,tn, will be the one for which the event 


F= {N(t1) = 1} {N(t2) — N(t1) = 1}... {N (tn) — N(tn-1) = 1} 


(5.54) 

occurs. Let 

G= {N(t1) > 1} N{N(t2) = 2}N...N{N(tnr) = n} (5.55) 
We have 
P[T, < t1,To < ta... ,Tn < tn, N(t) =] 

= P[FN{N(t) = n}] + P[GN{N(t) = n}] 

= P[FN{N(t) — N(tn) = 0}} + P[EN{N(t) = n}] 

= Atye7*4! x Il N(te — th_r)eT te) x em AE tn) 

+P[Gn (Nit) =n}] 
= A"tye"*" [l(t —th-1) + PIGN{N(t) = n}] (5.56) 
k=2 


It follows that 


P[T, <t1,T2 $ ta,..- Ta Stn | N(t) =n] 


_ PT, <t1,T2 < te,... ,Tn < tn, N(t) =n) 
~ P[N(t) =n] 


_ Attre™** [Tkeo(te — th—1) 


(t)"e—*/nl + P[G | N(t) =n] (5.57) 


Since at least one t; is not present in the term P[G | N(t) = n], we may write 
that 


ST, ,To,... Tn |N(t) (t1, #2, +--+ stn | n) 
Ot, Ot2 --- Oty t”/n! 
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! 
== for 0 <t, <tg<...<tn St (5.58) 


We have proved the following proposition. 


Proposition 5.1.6. Let {N(t),t > 0} be a Poisson process with rate A. Given 
that N(t) = n, the n arrival times of the events, T,,..., Tn, have the joint 
probability density function given by the formula (5.58). 


Remarks. i) Another (intuitive) way of obtaining the joint density function of 
the random vector (T),... ,T,), given that N(t) =n, is to use the fact that, 
for an arbitrary event that occurred in the interval (0,¢], the arrival time of 
this event has a uniform distribution on this interval. We can consider the n 
arrival times of the events as being the values taken by n independent U(0, t| 
random variables U; placed in increasing order (since T; < Ty <... < Th). 
There are n! ways of putting the variables U,, ..., Un in increasing order. 
Moreover, by independence, we may write, for 0 < t) < tg <...<tn<t 
that 


Ce ee [] f.tto = ls-= (5.59) 


Thus, we retrieve the formula (5.58): 


ni 


Ft,,...,Tajn(e) (ti, ++ ta | n) = nl fu,,... un (tis--. stn) = ar (5.60) 


if0<t) <te<...<ty <t. 


ii) When we place some random variables Xj, ..., Xn in increasing order, 
we generally use the notation X(1),... ,X(n), where XQ) < Xj) ift <j. In 
the continuous case, we always have that X(;) < X(j) if i < j. The variables 
placed in increasing order are called the order statistics of X,,...,Xn. We 
have that X(1) = min{X1,...,Xn} and X(,) = max{X),... , Xn}. 


Let us now return to the problem of decomposing the events of a Poisson 
process, {N(t),t > 0}, into two or more types. 


Proposition 5.1.7. Suppose that an event of a Poisson process with rate , 
{N(t),t > 0}, that occurs at time s is classified, independently from the other 
events, of type i with probability p;(s), where i =1,... ,j and S°}_, pi(s) = 1. 
Let N,(t) be the number of type i events in the interval [0,t]. The Ni(t)’s are 
independent random variables having Poisson distributions with parameters 


t 
Ai(t) = | pi{s) ds fori =1,2,...,9 (5.61) 
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Proof. We know that the arrival time of an arbitrary event that occurred in 
the interval (0, ¢] has a uniform distribution on this interval. Consequently, we 
may write that the probability p; that this event is of type 7 is given by 


t 
1 
Pi - | pi(s)> ds for 1 = 152, cee Jj (5.62) 
0 


Proceeding as in the proof of Proposition 5.1.2, we find that the random 
variable N;(t) has a Poisson distribution with parameter 


t 
Ai(t) = Apt = af pi(s) ds fort =1,2,...,9 (5.63) 
0 


and that the variables Ni(t), No(t), ..., Nj(t) are independent. 0 


Remark. The random variables N;(t) have Poisson distributions. However, 
the stochastic processes {N;(t),¢ > 0} are not Poisson processes, unless the 
functions p;(s) do not depend on s, for all 7. 


Example 5.1.3. Customers arriving at a car dealer (open from 9 a.m. to 9 p.m.) 
can be classified in two categories: those who intend to buy a car (type I) and 
those who are just looking at the cars or want to ask some information (type 
II). Suppose that 


1/2 from 9 a.m. to 6 p.m. 


P[Customer ts-of type 4] = { 1/4 from 6 p.m. to 9 p.m. 


independently from one customer to another and that the arrivals constitute 
a Poisson process with rate » per day. 


(a) Calculate the variance of the number of type I customers arriving in one 
day if X = 50. 

(b) Suppose that the average profit per car sold is equal to $1000 and that 
A = 10. What is the average profit for the dealer from 9 a.m. to 6 p.m. on a 
given day, knowing that at least two cars were sold during this time period? 


Solution. (a) Let N;(t) be the number of type I customers in the interval [0, t], 
where ¢ is in (opening) hours. We can write that 


12 


21 18 4 214 
[ p(s) as = [ pase | ~ ds = 21/4 
9 9 2 is 4 


We seek V[N;(12)] = 21.875. 


(b) Let Ni(t) be the number of sales in ¢ days, from 9 a.m. to 6 p.m. The 
process {Nj(t),£ > 0} is a Poisson process with rate 


N7(12) ~ Poi (F i pr(s) as) = Poi(21.875) 


because 
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9 1 
Nei 
oreo ae 


We are looking for 1000E[Nj(1) | Ni (1) > 2] := 1000x. We have 


= 3.75 


E(N,(1)] = 0 + ELNi(1) | Na(1) = 1PIN(1) = 1) + 2P IN) > 2 
It follows that 


3.75 =1-3.75e975 4 2 [1 —@3-Ty 4 3.75)| 


We find that x ~ 4.12231, so that the average profit is given by (approxi- 
mately) $4122.31. 


Proposition 5.1.8. Let {N,(t),t > 0} and {No(t),t > 0} be independent 
Poisson processes, with rates 1 and Az, respectively. We define the event 
Friny = 1 events of the process {N,(t),t > 0} occur before nz events of the 
process {N2(t),t > 0} have occurred. We have 


P|Fni,n2] = P[X > mi], where X ~ B(n =n t+n-1,p:= ws) 


(5.64) 
That is, 
PIF = ne ny tng-1 Ay i hy nitn2—1-i 
m2 = 4 Ay tAe2 Ay + A2 
(5.65) 


Proof. Let N(t) := Ni(t)+ No(t), and let E; be the random experiment that 
consists in observing whether the jth event of the process {N(t),t > 0} is an 
event of the process {N,(t),¢ > 0} or not. Given that the Poisson process has 
independent and stationary increments, the Ej’s are Bernoulli trials, that is, 
independent trials for which the probability that the jth trial is a success is 
the same for all 7. Then the random variable X that counts the number of 
successes in n trials has, by definition, a binomial distribution with parameters 
n and (see Prop. 3.3.4) 


Ai 
Ai +A2 


p= PIT); < T,1] => (5.66) 
where T; ~ Exp(A;) and Tz; ~ Exp(A) are independent random variables. 
Since Fy, n, occurs if and only if there are at least n, events of the process 
{Ni (t),t > 0} among the first n, +n —1 events of the process {N(t),t > 0}, 
we obtain the formula (5.65). O 
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Example 5.1.4. Let {X(t), 
ob 


< t < 1} be the stochastic process defined from 
a Poisson process {N(t) 0 


0 
> 0} with rate A as follows: 
X(t)=N(t)-tN(1) forO<t<1 


Note that, like the Brownian bridge (see p. 196), this process is such that 
X (0) = X(1) =0. We calculate 


E[X(t)] = E[N(t) — tN(1)] = ELN(t)] — tE[N(1)] = At — t(A- 1) = 0 
for all t € [0,1]. It follows that if 0 < t; <1, fori = 1,2, then 


Cx (ti, ta) = EB [X(t1)X (t2)] — 0 
= E{[N(t1) — ti N(1)] [N(ta) — taN(1)]} 
= E[N(ti)N(t2)] — 1B [N(1)N(t2)] — teB [N(ti1)N(1)] 
+ tyteE(N?(1)] 
= dtyto + A min{ty, to} — ty (A?t2 + Ate) — te (A7t1 + Ata) 
+ tyt2 (A7 +A) 
= A min{ty, to} = Atte 


where we used the formula 
Rn(ti, te) = Cy (th, ta) + E[N(t1)|E[N(t2)] = A min{ty, to} + d* tite 


We can generalize the process above by defining {X(t),0 < t < c} by 
X(t) = N(t) - ~N(c) for0<t<e 


where c is a positive constant. 


Example 5.1.5. We know that the Poisson process is not wide-sense stationary 
(see p. 232). On the other hand, its increments are stationary. Consider the 
process {X(t),t > 0} defined by 


X(t)=N(t+c)—N(t) fort >0 


where c is a positive constant and {N(t),t > 0} is a Poisson process with rate 
A. Note that X(0) = N(c) ~ Poi(Ac). Thus, the initial value of the process is 
random. We have 


N(t+c)—N(t)~ Poi(Ac) => E[X(t)]}=Ac for allt >0 


Next, by using the formula for Ry(ti,t2) in the preceding example, we 
calculate (for s,t > 0) 


Rx (t,t +8) := E[{N(t+c) — N(t)}{N(t+s+c) —-N(t+s)}] 
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E[N(t+c)N(t+s+c)] — E[N(t+c)N(t+s)| 

— E[N(t)N(t+s+c)] + E[N(t)N(t + 8)] 

Al(t +e) —t — min{c,s} —t+ ¢] 

+2 [(t+e)(t+s+c)—(t+e\(t+s)-—tt+s+oe)+t(t+s)] 
= A(e— min{e, s}) + A?e? 

It follows that 


Cx (t,t +s) := Rx(t,t+s) — E[X()|E[X(t + s)] 
= d(c— min{e, s}) + \%c? — (Ac)? 


0 ife<s 
= A(c— min{e,s}) = ae ife>s 


We can therefore write that Cx (t,t +s) = Cx(s). Thus, the stochastic 
process {X(t),t > 0} is wide-sense stationary (because E[X(t)] = Ac), which 
actually ca from the stationary increments of the Poisson process. More- 
over, the fact that the autocovariance function C’x(s) is equal to zero, for 
c S 8, is a consequence of the independent increments of {N(t),t > 0}. 


lI 


Example 5.1.6. We suppose that the number N(t) of visitors of a certain Web 
site in the interval [0,¢] is such that {N(t),t > 0} is a Poisson process with 
rate A per hour. Calculate the probability that there have been more visitors 
from 8 a.m. to 9 a.m. than from 9 a.m. to 10 a.m., given that there have been 
10 visitors from 8 a.m. to 10 a.m. 


Solution. Let N; (respectively, No) be the number of visitors from 8 a.m. 
to 9 a.m. (resp., from 9 a.m. to 10 a.m.). We can assert that the random 
variables Ny ~ Poi(5) and N2 ~ Poi(5) are independent. Furthermore, we 
have N; | {Ni + No = 10} ~ B(n = 10,p = 1/2), for i = 1,2. We seek the 
probability x := P[N; > Ng | Ni+N2 = 10]. By symmetry, we can write that 


10 
=2+ P(N; = No|N,+No=10) +2 = 264+ ("pyar ~ 2x + 0.2461 
Then z ~ 0.3770. 


Example 5.1.7. We define M(t) = N(t) — (t?/2), for t 
t > 0} is a Poisson process with rate \. Calculate P[1 
S, := min{t > 0: M(t) > 1}. 


Solution. The increments of the Poisson process being independent and sta- 
tionary, we can write that 


Pil <8, < v2] 
P[N(1) = 0, N(vV2) — N(1) > 2] + PIN(1) =1,N(V2) — N(1) > J] 
“f= eM (1 4 (V2 - )A)] + r\em* 1 — eM] 


where {N(t) 


> 0, 
< 8; < v3], where 


Il 
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Fig. 5.2. Example of a trajectory of a telegraph signal. 


5.1.1 The telegraph signal 


As we did with the Brownian motion and in Examples 5.1.4 and 5.1.5, we can 
define stochastic processes from a Poisson process. An interesting particular 
transformation of the Poisson process is the telegraph signal {X(t),t > O}, 
defined as follows: 


Age. 
oan (5.67) 


An example of a trajectory of a telegraph signal is shown in Fig. 5.2. 


Remark. Note that X(0) = 1, because N(0) = 0. Thus, the initial value of the 
process is deterministic. To make the starting point of the process random, 
we can simply multiply X(t) by a random variable Z that is independent of 
X(t), for all t, and that takes on the value 1 or —1 with probability 1/2. It is 
as if we tossed a fair coin at time t > 0 to determine whether X(t) = 1 or —1. 

The process {Y(t),t > 0}, where Y(t) := Z- X(t), for all t > 0, is called 
a random telegraph signal. We may write that Z = Y(0). Moreover, to be 
precise, we then use the expression semirandom telegraph signal to designate 
the process {X(t),t > O}. We already encountered the random telegraph 
signal in Example 2.3.2. 


To obtain the distribution of the random variable X(t), it suffices to cal- 
culate 


(2k)! 
k=0 
At —Mt —~2\t 
pee Te Lee ee: EE) 
2 2 
because 
At —At co 2k 
e+e (At) 
coshaAt := 5 = S- (5.69) 
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where “cosh” denotes the hyperbolic cosine. It follows that 


1 —2At Tes —2At 
P[X(t) = —1] =1- a = (5.70) 


In the case of the process {Y(t),¢ > 0}, we have 


Bs) 
= 
f 
| 


= P[Z-X(t)=] 

= P[X(t)=1|2Z=1)P[Z=1) 
+ P[X(t) =-1|Z=—-1)P[Z = -1] 

me 5 (PIX(t) = 1) + P[X(t)=-1]} = : (5.71) 

Thus, P[Y(¢) = 1] = P[Y(t) = —-1] = 1/2, for all t > 0, so that E[Y(t)] = 0, 

whereas 


B[X(t)] = 1 =S— + (-1)- 5 


V¥t>0 (5.72) 


To obtain the autocorrelation function of the (semirandom) telegraph sig- 
nal, we make use of the definition X(t) = (—1)%™. If s > 0, we may write 
that 


Rx (t,t +8) := E[X(t)X(t+s)]=E [(-1)" (yr) 
= —F | (—1)24@) (ayers) (0) =F [(-1yer)- 0) 
=E (1% = E[X(s)] = 7?" (5.73) 


where we used the fact that the increments of the Poisson process are station- 
ary. We then have 


Cx (t,t + s) = e72A8 = e7 2Ate—2A(tts) — e72h8 (1 — es) V s,t > 0 
(5.74) 


Finally, since 


we find that 
Cy(t,t+s) = Ry(t,t+s) =Rx(t,t+s)=e* Vs,t>0 (5.76) 


We deduce from what precedes that the process {Y (t),t > 0} is wede-sense 
stationary. As we mentioned in Example 2.3.2, it can even be shown that it is 
strict-sense stationary. On the other hand, {X(t),t > 0} is not WSS, because 
its mean depends on t (and Cx (t,t +s) 4 Cx(s)). Furthermore, in Example 
2.3.2, we showed that the random telegraph signal is a mean ergodic process. 
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5.2 Nonhomogeneous Poisson processes 


In many applications, it is not realistic to assume that the average arrival 
rate of events of a counting process, {N(t),t > 0}, is constant. In practice, 
this rate generally depends on the variable t. For example, the average arrival 
rate of customers into a store is not the same during the entire day. Similarly, 
the average arrival rate of cars on a highway fluctuates between its maximum 
during rush hours and its minimum during slack hours. We will generalize the 
definition of a Poisson process to take this fact into account. 


Definition 5.2.1. Let {N(t),t > 0} be a counting process with independent 
increments. This process is called a nonhomogeneous (or nonstationary ) 
Poisson process with intensity function \(t) > 0, fort > 0, if N(0) =0 
and 

i) PIN(t +6) — N(t) = 1] = A(t)6 4+ (6), 

ti) P[N(t + 6) — N(t) > 2] = o(6). 


Remark. The condition i) implies that the process {N(t),t > 0} does not have 
stationary increments unless \(t) = \ > 0. In this case, {N(t),t > 0} becomes 
a homogeneous Poisson process, with rate . 


As in the particular case when the average arrival rate of events is constant, 
we find that the number of events that occur in a given interval has a Poisson 
distribution. 


Proposition 5.2.1. Let {N(t),t > 0} be a nonhomogeneous Poisson process 
with intensity function X(t). We have 


N(s+t)—N(s) ~ Poi(m(s+t)—m(s)) Vs,t>0 (5.77) 
where 
m(s) := [ (rT) dr (5.78) 
0 
Proof. Let 
Pn(s,t) := P[N(s+t)—N(s) =n] forn=0,1,2,... (5.79) 


Using the fact that the increments of the process {N(t),t > 0} are indepen- 
dent, and using the two conditions in the definition above, we may write, for 
n=1,2,..., that 


Pn(s,t +) 
= P(N(s +t) — N(s) =n, N(s+t+6) —N(s+t) =0] 
+ P[N(s+t)—N(s) =n—1,N(s+t+6) —N(s +t) =1]+4 0(6) 
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= Pn(s,t)[1 — A(s + t)6 + 0(5)] + pn_1(s,t)[A(s + £)5 + 0(6)] + 0(5) 
from which we find that 
Pn(s,t +4) — pr(s,t) = A(s + t)6[pr_1(s,t) — pn(s,t)] + 0(5) (5.80) 


Dividing both sides by 6 and taking the limit as 6 decreases to zero, we obtain 


< palsst) = X(s + t)[pr—1(s,t) — Pr(s,t)] (5.81) 


When n = 0, the equation above becomes 


5 po(sst) = —As + t)p0(t (5.82) 


The variable s may be considered as a constant. Therefore, this equation is 
a first-order, linear, homogeneous ordinary differential equation. Its general 
solution is given by 


Po(s,t) = co exp {- ) ™ A(T) ar} (5.83) 


where cp is a constant. Making use of the boundary condition po(s,0) = 1, we 
obtain cp = 1, so that 


s+t 
po(s,t) = exp {-/ X(T) ar} = em(s)—m(st+t) for 5,¢ >0 (5.84) 
s§ 
Substituting this solution into Eq. (5.81), we find that 


O 
prise) = X(s +t) jesgenen) — pr(s,t)| (5.85) 
We may rewrite this equation as follows: 

a m(s)—m(s+t) 0 

—p1(s,t) = le - pr(s,t)| = [m(s + t) — m(s)| (5.86) 

Ot Ot 
We easily check that the solution of this nonhomogeneous differential equation, 
which satisfies the boundary condition p;(s,0) = 0, is 

pi(s, t) = e™)-™(s+) Im(s +t) —m(s)] for s,t > 0 (5.87) 
Finally, we can show, by mathematical induction, that 


ee n 
pr(s,t) = em(s)—m(s+t) [m(s + t) — m(s)]” Vs,t>O0andn=0,1,...0 


(5.88) 


n! 
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Remarks. i) The function m(t) is called the mean-value function of the process. 


ii) Suppose that A(t) is a twice-differentiable function such that A(t) < A, 
for all ¢ > 0. We can then obtain a nonhomogeneous Poisson process with 
intensity function A(t) from a homogeneous Poisson process {N;(t),t > 0}, 
with rate \, by supposing that an event that occurs at time t is counted with 
probability A(t) /A. Let 

F = exactly one of the events that occur in the interval (t, +46] is counted, 
and let 

N(t)= the number of events counted in the interval (0, ¢]. 
We have (see Prop. 5.1.7) 


P[N(t +6) - = SP [{mi(t+ 6) - i(t) = kK} NF] 
k=1 
6 
[Ad + 0(6)] [ me du + o(d) 
KieeGy = + ) 5 403) (5.89) 


for some c € (0,1), by the mean-value theorem for integrals. Since 
A(t + cd) = A(t) + cdA’(t) + 0(5) (5.90) 


we may write that 


PIN(t + 8) — N(t) = 1] = Pd + (4) MOF OA #0(8) 
= X(t)d + o(6) (5.91) 
iii) If we assume that 
i’ A(t) dt = 00 for all to € (0, 00) (5.92) 
to 


then the probability that at least one event will occur after to is equal to 1: 


Jim, P(N (to + 8) — N(to) > 1] = 1 ~ Jim PIN (to + 8) — N(to) = 0 


tots 
=1-—- lim exp {- [ A(t) ar} 
s—0co to 


=1-0=1 (5.93) 


Note that the formula (5.92) is valid when X(t) = A > 0. 


Let J; be the random variable that denotes the arrival time of the first 
event of the process {N(t),t > 0}. We will now calculate the distribution of 
Ti, given that N(t) = 1, as we did in the case of the homogeneous Poisson 
process (see Prop. 5.1.5). 
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Proposition 5.2.2. Let {N(t),t > 0} be a nonhomogeneous Poisson process 
with intensity function A(t). The probability density function of the random 
variable S := T, | {N(t) = 1} is given by 


fs(s) = ney forO0<s<t (5.94) 
m 
Proof. If 0 < s < t, we may write that 


Presing ai at as Ne 


PING) =] 
ind, (m(s)e~™()) e7~m(t)+m(s) a m(s) 
~ mite" ~ m(t) 
Since 
mts) = =f Aw) du = Xs) (5.95) 


we obtain the formula (5.94). 0 


Remark. The formula for the distribution function of the arrival time S of the 
single event that occurred in an arbitrary interval (7,7 + t] is 


m(s) — m(r) 


a apr a ae Si dy A rt s< t 5.96 
Fs(s) +e) @) forO<r<s<rt (5.96) 
so that 
X(s) 
fs(s) EVP) forO<r<s<rt ( ) 


where S := T | {N(r +t) — N(r) = 1} and T is the arrival time of the first 
event of the nonhomogeneous Poisson process in the interval (7,7 + ¢]. 


Example 5.2.1. Let {N(t),t > 0} be a nonhomogeneous Poisson process with 
intensity function A(t) > 0, for t > 0. We set 


M(t)=N(m7‘(t)) for allt >0 


where m7!(t) is the inverse function of the mean-value function of the process. 


Remark. This inverse function exists, because we assumed (in this example) 
that the intensity function A(t) is strictly positive, for all t, so that m(t) is a 
strictly increasing function. 


We then have 
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Moreover, we calculate, for all 7,t > 0 and n = 0,1,..., 
PIM(r +t) - M(t) =n] = P[N(m7(r +2)) —N (m7*(7)) =n] 
= P [Poi (m(m71(r + t)) —m(m7"(r))) =n] 
= P[Poi(r +t —T) =n] = P [Poi(t) = n] 
Finally, since the function m~!(t) is strictly increasing, we may assert that 
the stochastic process {M(t),t > 0}, like {N(t),¢ > 0}, has independent in- 


crements. We then deduce, from what precedes, that {M(t),t > 0} is actually 
a homogeneous Poisson process, with rate A = 1. 


5.3 Compound Poisson processes 


Definition 5.3.1. Let X1,Xo,... be independent and identically distributed 
random variables, and let N be a random variable whose possible values are 
all positive integers and that is independent of the X;,’s. The variable 


N 
Suis So Xe (5.98) 
k=1 


is called a compound random variable. 


We already gave the formulas for the mean and the variance of Sy [see 
Eqs. (1.89) and (1.90)]. We will now prove these formulas. 


Proposition 5.3.1. The mean and the variance of the random variable Sn 
defined above are given, respectively, by 


N 
E|\S° X,| = BINJE[Xy] (5.99) 
k=1 
and 
N 
V Is. x = E(N|V[Xi] + VIN|(E[X1])? (5.100) 
k=1 


Proof. First, we have 
E ba x = S~ E[X,] © nE[X1] (5.101) 
k=1 k=1 
Then, since N is independent of the X;,’s, we may write that 


N n 
So XK |N = | =E ps x = nE[X1] (5.102) 
k=1 =1 


E 
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so that 


N 
E Ds Ke 
k=1 


x| = NE[X,] (5.103) 
It follows that 
N N 
E [yo =E\E om 
k=1 k=1 


Remark. It is not necessary that the random variables X, be independent 
among themselves for the formula (5.99) to be valid. 


x|| = E[NE[X,]] = E[NJE[Xi] (5.104) 


Next, proceeding as above, we find that 


N 
Be 


N 
V|So XIN = | ie nVIX,] => V n| = NV[X,] (5.105) 
k=1 k=1 
With the help of the formula (see p. 28) 
V[Sy] = E[V[Sw | N]] + V[E[Sw | NJ] (5-106) 


we may then write that 


V|S_ X,| = BINV[X]) + VINE[XA] 
k=1 


= E[N]V[X\] + (21% ))?V[N] oO (5.107) 


Definition 5.3.2. Let {N(t),t > 0} be a Poisson process with rate X, and let 
X,,X2,... be random variables that are t.i.d. and independent of the process 
{N(t),t > 0}. The stochastic process {Y(t),t > 0} defined by 


N(t) 
Y(t)= 5X, Vt>0 (and Y(t) =0 if N(t) =0) (5.108) 
k=1 


is called a compound Poisson process. 


Remarks. i) This is another way of generalizing the Poisson process, since 
if the random variables X, are actually the constant 1, then the processes 
{Y(t),t > 0} and {N(t),t > 0} are identical. 

ii) A Poisson process, {N(t),¢ > 0}, only counts the number of events that 
occurred in the interval [0,¢], while the process {Y (t),t > 0} gives, for exam- 
ple, the sum of the lengths of telephone calls that happened in [0,¢], or the 
total number of persons who were involved in car accidents in this interval, 
etc. Note that we must assume that the lengths of the calls or the numbers 
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of persons involved in distinct accidents are independent and identically dis- 
tributed random variables. We could consider the two-dimensional process 
{(N(t), Y(t)),t > 0} to retain all the information of interest. 


Using Proposition 5.3.1 with Sy = Y(t) and N = N(t), we obtain 
E[Y (t)| = E[N(t)]E[X1] = AtE[X4] (5.109) 
and 
VIY (t)] = E[N(t)|V[X4] + VIN ())(E[X4])? 
= M (V[Xq] + (E[X1])?) = AtE[X} | (5.110) 


We can easily calculate the moment-generating function of the random 
variable Y(t). Let 


M,(s) = Mx,(s) := E [e***] (5.111) 
We have 


ag [ew a] = B|e Het Xw0)] op [E [erXt--+Xmco)] |] 


ind. oe 3(Xi+.. a] eel iid. Sli (ste OO” 
=0 ‘ n=0 


n 
= e*eMi(s)At — exp{At[My(s) — 1]} (5.112) 


~ 
5 


This formula enables us to check the results obtained above. In particular, 
we have 


EIY(t)) = © Myw(s)|,_g = Mv ty (s)tMi(s)| 


ds s=0 
= My (¢)(0)AtMj{(0) = 1AtE[X4] = AtELX)| (5.113) 
When X; is a discrete random variable, whose possible values are 1,2,... ,j, 
we may write that 
j 
Y(t) = 5_iNi(t) (5.114) 
i=l 


where N;,(t) is the number of random variables X;, (associated with some 
random events) that took on the value i in the interval [0,¢]. By Proposition 
5.1.2, the processes {Nj(t),t > 0} are independent Poisson processes with 
rates Apx, (2), for? =1,... ,7. 

Remarks. i) This representation of the process {Y(t),t > 0} can be generalized 
to the case when X, is an arbitrary discrete random variable. 
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ii) The moment-generating function of the random variable Y(t) in the case 
above becomes 


My @)(s) = = expat 3o le" 1) px, ( ‘} (5.115) 


Since limy_... N(t) = 00, we deduce from the central limit theorem the 
following proposition. 


Proposition 5.3.2. Fort sufficiently large, we may write that 
Y(t) = N (AtE[X]], AtE[X7]) (5.116) 


Remark. For the approximation to be good, the number of variables in the 
sum must be approximately equal to 30 or more (or maybe less), depending 
on the degree of asymmetry of (the distribution of) the random variable X, 
with respect to its mean. 


Finally, let {Yi (t),t > 0} and {Y2(t),t > 0} be independent compound 
Poisson processes, defined by 


N;(€) 
=S° Xin Vt2>0 (and ¥;,(t) =0 if Ni(t) = 0) (5.117) 
k=1 
where {N,(t),¢ > 0} is a Poisson process with rate \;, for 1 = 1,2. We know 
that the process {N(t),t > 0}, where N(t) := N\(t) + No(t) Vt > 0, isa 
Poisson process with rate A := 1 + Az (because the two Poisson processes are 
independent; see Prop. 5.1.1). Let X; be the random variable associated with 
the kth event of the process {N(t),t > 0}. We may write that 


X 1,4 with probability p := ——— 
xX, & Ar + Ag (5.118) 
X2,, with probability 1—p 


(That is, X; has the same distribution as X1,, (respectively, X24) with prob- 
ability p (resp., 1 — p).) Thus, we have 


P[Xk Sa] = P(X, S tlp+ P[X24 < a](1 —p) (5.119) 


Since the random variables X,, X2,... are i.i.d., and are independent of the 
Poisson process {N(t),t > 0}, we may assert that the process {Y(t),t > 0} 
defined by 


Y(t) =Yi(t)+Y¥o(t) fort >0 (5.120) 


is a compound Poisson process as well. 
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Example 5.3.1. Suppose that the customers of a certain insurance company 
pay premiums to the company at the constant rate a (per time unit) and 
that the company pays indemnities to its customers according to a Poisson 
process with rate \. If the amount paid per claim is a random variable, and if 
the variables corresponding to distinct claims are independent and identically 
distributed, then the capital C(t) at the company’s disposal at time ¢ is given 
by 
C(t) = C(0) + at — Y(t) 


where {Y(t),¢ > 0} is a compound Poisson process (if, in addition, the indem- 
nity amounts do not depend on the number of indemnities paid). An important 
question is to be able to determine the risk that the random variable C(t) will 
take on a value smaller than or equal to zero, that is, the risk that the com- 
pany will go bankrupt. If a < Ay, where ys := E[X,] is the average indemnity 
paid by the company, then we find that the probability that it will eventually 
go bankrupt is equal to 1, which is logical. 


5.4 Doubly stochastic Poisson processes 


In Section 5.2, we generalized the definition of a Poisson process by allowing 
the average arrival rate of events of the process to be a deterministic function 
A(t). We will now generalize further the basic Poisson process by supposing 
that the function A(t) is a random variable A(t). Thus, the set {A(t),t > 0} 
is a stochastic process. For this reason, the process {N(t),t > 0} is called a 
doubly stochastic Poisson process. First, we consider the case when the random 
variable A(é) does not depend on t. 


Definition 5.4.1. Let A be a positive random variable. If the counting process 
{N(t),t > 0}, given that A = A, is a Poisson process with rate X, then the 
stochastic process {N(t),t > 0} is called a conditional (or mixed) Poisson 
process. 


Proposition 5.4.1. The conditional Poisson process {N(t),t > 0} has sta- 
tionary, but not independent increments. 


Proof. Consider the case when A is a discrete random variable whose possible 
values are in the set {1,2,...}. We have 


P(N(r +t) — N(r) =n] = 5> PIN (r +t)-N(r) =n | A=K] = palk) 


k=1 
= Soe BP ACh) (5.121) 
k=1 


Since N(0) = 0, for any value of A, we conclude that 
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P[N(r +t) — N(r) =n] = P[N(t) =n) for all 7,t > 0 andn=0,1,... 
(5.122) 


Thus, the increments of the process {N(t),t > 0} are stationary. 
Next, using the formula (5.121) above, we may write that 


_ PIN(t) =n| A= j)PIA = J] 
P[N(t) =n) 
_ __ e FGt)"/ni}pald) 
per OW (kt) /nl|pa(k) 
a = Og A) 
= $e (5.123) 


Therefore, the number of events that occurred in the interval [0,t] gives us 
some information about the probability that the random variable A took on 
the value j. Now, the larger j is, the larger is the expected number of events 
that will occur in the interval (t,t + 7], where 7 > 0. Consequently, we must 
conclude that the increments of the process {N(t),t > 0} are not independent. 
QO 


Remarks. i) In the case of the homogeneous Poisson process, with rate A > 0, 
we have that pa(A) = 1, so that 


e-MI(\E)" /nl] x 1 


BAS AUN SO Coan ial 


=1 forn=0,1,... (5.124) 


ii) The proposition above implies that a conditional Poisson process is not a 
Poisson process unless A is a constant. 


iii) If the parameter of the homogeneous Poisson process {N(t),t > 0} is 
unknown, we can estimate it by taking observations of the random variable 
N(t), for an arbitrary t > 0. Suppose that we collected n (independent) ob- 
servations of N(t). Let X be the average number of events that occurred in 
the interval [0,t]. That is, 


mr 
X:= Dekat Xk (5.125) 
n 


where X;, is the kth observation of N(t). The best estimator of the parameter 
A is given by 
(5.126) 


Thus, the larger the number of events that occurred in the interval (0, ¢] is, 
the larger the estimated value of is. However, once this parameter has been 
estimated, we start anew and we assume that the increments are independent. 
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iv) When A is a continuous random variable, we find that 


gh VF Ar) 


fain |) = Toe" fal) da VA>0 (5.127) 
Given that 
E(N(t)| A] =4At and V{[N(t)| A] = At (5.128) 
we calculate 
EIN(t)| = EIBIN(t) | Al] = tE[4] (5.129) 


and 


VIN(t)] = ELVIN (¢) | Al] + VIELN(E) | 4] 
= E[At| + V[At] = tE[A] + VA] (5.130) 


Example 5.4.1. Suppose that the rate of a Poisson process is a continuous 
random variable A such that 


a 
fal) = 53 ADI 


It can be shown that 


P[N(t) > n] = / 5 Pi[A> rite A" dd 


Use this formula to calculate the probability that there will be more than two 
events during an arbitrary time unit. 


Solution. First, we calculate 


Plu>rl= [oS ae= 5 for \ >1 
as Xr x3 ~ 2 a 
We seek 

: ? POT a agent 1 ft 1 
N = iene Se Bane r=5/ dv —* dy Zocl 
P{N(1) > 2} [ve 5 an | ye 5 d 5 I, € + 5e 


Doing the integral above by parts, we find that 


P[N(1) > 2] = 5 (2—5e7! + e71) =1-2e7' ~ 0.2642 
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Example 5.4.2. If A has a geometric distribution, with parameter p € (0,1), 
then we deduce from the formula (5.121) that 


Pr Pp, wd p, d r p tr 
[N(t) = 1] at rs ; = (rs) 1 


Example 5.4.3 (The Pélya' process). Suppose that A has a gamma distri- 
bution, with parameters a = k € N and @ > 0. We have 


P[N(t) = nl 


P(N(t) =n | A= Alfa(A) da 


oe) ent Qt)” (ge-P) ann 
tr Bk 


n! 
WO U—(tHB)Ayntk-1 
eae =(t+ n+k— 
=F Eon | : ‘ aA 


dr 


i 
i 


nr ky 66 
a im merare f ent yttk-l dx (with r= (t + B)d) 
a 1 
- nl (R— DI + Byeret @ +) 
oe 1 
= Gil pee eo 
= es "pk —p)” forn=0,1,... 


where 
Pp —— a#. 
“ t+2 
1 George Pélya, 1887-1985, was born in Hungary and died in the United States. 


He contributed to several domains of mathematics, including probability theory, 
number theory, mathematical physics, and complex analysis. 
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We say that N(t) has a negative binomial, or Pascal? distribution, with 
parameters k and p. Furthermore, the process {N(t),t > 0} is called a negative 
binomial process, or a Polya process. 


Remark. There exist various ways of defining a negative binomial distribution. 
The version above generalizes the geometric distribution when we define it as 
being a random variable that counts the number of failures, in Bernoulli trials, 
before the first success, which can be checked by setting k = 1. Thus, here N(¢) 
would correspond to the variable that counts the number of failures before the 
kth success. Note also that if k = 1, then A has an exponential distribution 
with parameter /. 


Definition 5.4.2. Let {A(t),t > 0} be a stochastic process for which A(t) 
is nonnegative, for all t > 0. If, given that A(t) = X(t), for allt > 0, the 
counting process {N(t),t > 0} is a nonhomogeneous Poisson process with 


intensity function X(t), then {N(t),t > 0} ts called a doubly stochastic 


Poisson process or a Cox® process. 


Remarks. i) If the random variable A(t) does not depend on ¢, we retrieve 
the conditional Poisson process. Moreover, if A(t) is a deterministic function 
A(t), then {N(t),t > 0} is a nonhomogeneous Poisson process, with intensity 
function A(t). The doubly stochastic Poisson process thus includes the ho- 
mogeneous, nonhomogeneous, and conditional Poisson processes as particular 
cases. 


ii) The doubly stochastic Poisson process is not a Poisson process unless the 
random variable A(t) is equal to the constant » > 0, for all ¢ > 0. 


iii) The process {A(t),t > 0} is called the intensity process. 


We can write, for all tg > t, > 0, that 
N(t2) — N(t1) | {A(£) = A(t) V t > 0} ~ Poi(m(t2) — m(t1)) (5.131) 


where 
t 
m(t) =| A(s) ds (5.132) 
0 
It can also be shown that 


k 
tg A(t mi i: A(t) dt) 
P[N(tz) — N(t1) =k | {A(t),0 <t <t < to}] = ef 4 eae | an 
(5.133) 
fork =0,1,.... 


? See p. 135. 
3 Sir David Cox, professor at the University of Oxford, in England. 
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To simplify the writing, set 


N(t) | {A(s),0 < s <t} = M(t) | A(O,t) (5.134) 
We then have 
E(N(t) | A(0,t)] = [ a) ds (5.135) 
so that 
E[N(t)| = E[E[N(t) | A(0,t)]] = E [fae yas] = [Bia E[A(s)| ds (5.136) 


Similarly, we have 


E[N?(t) | A(0, t)] = [40 ds + ([ A(s) is) (5.137) 


2|(faoa) [=e[f [a0 _— 
ay fi EA) ] ds du (5.138) 


E(N?(t)] = E[E[N?(t) | A(0,t)]] -[ E{A(s)] ds +f / Ra(s,u) ds du 
(5.139) 


Since 


we may write that 


Example 5.4.4. Suppose that {A(t),t > 0} is a homogeneous Poisson process, 
with rate A > 0. Then 


E{A(t)] =At and Ry(s,u) = Amin{s,u} + A? su 


We calculate j j 
E(N(t)] =| As ds = ll 
0 2 


and 


E[N?(t)] = 258 = + [ [\ min{s, u} + \?su] ds du 


Lye e+ ff xmingsyu} ds du +9? 


U t4 
apa | sds+ [was deep or — 
2 0 0 uU 4 
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fo oP t4 
= Jap jae be 
2 2 3 e 4 
It follows that 


2 3 4 2\ 2 #28 
VIN(t)] = 5 ¢ AS + XS 2 (5) 2A (5 n 5) 


Example 5.4.5. lf {A(t), t > 0} is a geometric Brownian motion, then A(t) has 
a lognormal distribution (see p. 186), with parameters pt and ot. Its mean 
is given by 


1 
E{A(t)] = exp{ut + 57 th 
We then have 


: 1 1 1 
E(N(t)] = =¢7s\ ds = ——— t+-o07t} -1 
[N(t)} [ exptus+ 52 s} ds py igs PtH 57 }-]] 


if w+ $0? £0 (and E[N(t)] =¢ if p+ 40? = 0). 


5.5 Filtered Poisson processes 


After having generalized the homogeneous Poisson process in various ways, 
we now generalize the compound Poisson process. 


Definition 5.5.1. Let X;, Xo,... be random variables that are t.i.d. and in- 
dependent of the Poisson process {N(t),t > 0}, with rate \ > 0. We say that 
the stochastic process {Y(t),t > 0} defined by 


N(t) 
Y(t) = S> w(t,Te, Xz) Vt>0 (Y(t) =0 éf N(t) =) (5.140) 
kel 


where the T,’s are the arrival times of the events of the Poisson process and 
the function w(-,-,-) is called the response function, is a filtered Poisson 
process. 


Remarks. i) The compound Poisson process is the particular case obtained by 
setting w(t, T,, X~) = Xk. 


ii) We can say that the response function gives the value at time ¢ of a signal 
that occurred at time 7, and for which the quantity X; has been added to 
the filtered Poisson process. The random variable Y(t) is then the sum of the 
value at time ¢ of every signal that occurred since the initial time 0. 
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iii) An application of this type of process is the following: the variable Y(t) 
represents the flow of a certain river at time ¢ (since the beginning of the 
period considered, for example, since the beginning of the thawing period), 
the T;,’s are the arrival times of the precipitation events (snow or rain), and 
X, is the quantity of precipitation observed at time T;, and measured in inches 
of water or water equivalent of snow. In practice, the precipitation does not 
fall to the ground instantly. However, we must discretize the time variable, 
because the flow of rivers is not measured in a continuous way, but rather, in 
many cases, only once per day. Then, the index k represents the kth day since 
the initial time and X, is the quantity of precipitation observed on this kth 
day. A classic response function in this example is given by 


w(t, Te, Xk) = Xpe7"-™)/¢ fort > Th (5.141) 


where c is a positive constant that depends on each river and must be esti- 
mated. 


Suppose that we replace the random variable T; by the deterministic vari- 
able s in the function w(t, T,, X;). Let C,,(0) be the characteristic function 
of the random variable w(t, s, X,): 


C,,(8) (= Cu(6,t,8)) = E Giana! (5.142) 


Making use of Proposition 5.1.6, we can show that the characteristic function 
of Y(t) is given by 


t 
Cy (z)(9) = exp {-¥ + | Cw (9) as} (5.143) 
0 
from which we obtain the following proposition. 
Proposition 5.5.1. If E[w?(t,s,X;)] < co, then the mean and the variance 
of Y(t) are given by 
t 
E[Y(t)] = af E[w(t,s, Xx)] ds (5.144) 
0 
and 


V[Y(o)] = af Btw? sXe) ds (5.145) 


Remarks. i) When X, is a continuous random variable, the mathematical 
expectations E[w”(t,s, X,)], for n = 1,2, are calculated as follows: 


oO 


Efw"(t, 8, X)] = / sib doth Fy (ey ae (5.146) 


—oO 
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ii) It can also be shown that 


min{t1,t2} 
Cov|Y (#1), Y(t2)] = | Ejw(t1, 8, X,)w(te, s,X,~)] ds Vti,te 20 
0 
(5.147) 


Example 5.5.1. Suppose that the random variables X; have an exponential 
distribution with parameter yu in the formula (5.141). We can then write that 


Cy (0) = E [efeutton.Xs)| 5 ean —-E [efathesed xe) 


where 
g(t, s,c) = e t-9)/e 
We have 
E Eee = i” ef 99(t 8.0) je He dz = Sa 
0 bu — jOg(t, s,c) 


Next, we calculate 


t t t : D 
LU pe — jOg(t, s,c) + j8g(t, 8, c) 
C ayds= | oh as = [Re ae 
I wl) 0 H— J0g(t, 8,0) 0 ue — j0g(t, s,c) 


t s 
jOg(t, 8, c) 
=t+ [ ds 
0 UM j8g(t, s,c) 


Given that 


0 1 1 
Bg Its 8: ¢) = pens = Its) 


we may write that 


t . 
j0g(t, s,c) . s=t 
———_—-—__—— ds = —cln — 76 t,,C = 
i ee [u — J0g(t, 8,¢)]|5—o 


= —cln(u — j0) +eln(u — j0e7*/*) 
It follows that 


Cy (t)(9) = exp {at + Xt — cln(u — 76) + cin(u — joe-"/*)\\ 


€ ey 
~ \ p58 


Using this formula, we find that 


ElY(t)) = “ (1 —- er) and V[Y(t)] = Ac (a = ew He) 


ie 
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Remarks. i) In this example, we have 


Elw"(t, s, Xx)| —£ [xge-ne-2/¢| = ea nt-s)/¢ Rp [XP] 


= e-n(t-s)/eZ (m + 1) 
pr 
We then deduce from the formulas (5.144) and (5.145) that 


de 
Y(t)| =. fay (hax ts=r fe e-G-/ed gy — A (4 _ --t/e 
BY (t)| I d= ) 


and 


<oo forn=0,1,... 


t t 
2 
ViY(t)] = af E[w?(t, s, Xx)] ds = af e At s)/e_ ds 
0 0 


Ae —2t/c : 
saa re ) 


ii) We can check that the covariance of the random variables Y(t) and Y(t+r) 
is given by 


as above. 


Cov[Y(t), Y(t +7)] = ae ele (a _ ewe) Vt,r>0 


Finally, we can generalize the notion of filtered Poisson process by setting 


N(t) 
= > Wi(t,T.) Vt>0 (and Y(t) =0 if N(t) = 0) (5.148) 


where {W,(t, s),t > 0, s > 0} is a stochastic process (with two time parame- 
ters). We assume that the processes {W;(t,s),¢ > 0,s > 0} are independent 
and identically distributed and are also independent of the Poisson process 
{N(t),t > 0} (having rate A > 0). We say that {Y(t),t > 0} is a generalized 
filtered Poisson process. It can be shown that 


t 
EY @)| = af E{Wi(t,s)] ds and V/[Y(t) j= [ E[W7(t, s)] ds 
(5.149) 


5.6 Renewal processes 


An essential characteristic of the Poisson process, {N(t),t > 0}, is that the 
time between consecutive events is a random variable having an exponential 
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distribution with parameter A, regardless of the state in which the process 
is. Moreover, the random variables 7; denoting the time that {N(t),t > 0} 
spends in state i, fori = 0,1,..., are independent. 

We also know that the Poisson process is a particular continuous-time 
Markov chain. Another way of generalizing the Poisson process is now to 
suppose that the nonnegative variables 7; are independent and identically 
distributed, so that they can have any distribution, whether discrete or con- 
tinuous. 


Definition 5.6.1. Let {N(t),t > 0} be a counting process, and let 7; be the 
random variable denoting the time that the process spends in state i, for 
i = 0,1,.... The process {N(t),t > 0} is called a renewal process if the 
nonnegative variables T),7,,... are independent and identically distributed. 


Remarks. i) It is sometimes possible (see Ex. 5.1.2) to transform into a Poisson 
process a renewal process for which the time spent in the states 0,1,... does 
not have an exponential distribution. 


ii) We say that a renewal has occurred every time an event of the counting 
process takes place. 


iii) Some authors suppose that the random variables 79,71,... are strictly 
positive. 


iv) We can generalize the definition above by supposing that the random vari- 
able 79 is independent of the other variables, 71, 72,... but does not necessarily 
have the same distribution as these variables. In this case, {N(t),t > 0} is 
called a modified or delayed renewal process. 


The time T,, of the nth renewal of the (continuous-time) stochastic process 
{N(t),t > 0}, defined by [see Eq. (5.32)} 


n-1 
T=) a Pre Sl, (5.150) 
i=0 
satisfies the relation T, <t <= N(t) > n. By setting Tp = 0, we may write, as 
in Proposition 5.1.4, that 
N(t) = max{n > 0:7, < t} (5.151) 
Since 
P(N(t) =n] = P[N(t) > n] — PIN(t) > n+ 1] (5.152) 
we can state the following proposition. 


Proposition 5.6.1. The probability mass function of the random variable 
N(t) can be obtained from the formula 


PIN(t) =n] = PlTa <t]— Pian St] (5.153) 
fort >0 and forn=0,1,.... 
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In general, it is difficult to find the exact distribution function of the 
variable T,,. We have 


n-1 
Cr,(w) = Ble™] =n [E's] = 2 iT on] 
i=0 


n-1 n~1 
ind. U E [ie] = I] C,,(w) (5.154) 
i==0 i=0 


When 7; is a continuous random variable, the characteristic function C7, (w) 
is the Fourier transform of the probability density function fr, (t) of T,. We 
can then write that 


fr, (t) = fro(to) * fry (ti) * -.. * fray (tn-1) (5.155) 
where to +t, +...+tn-1 = t. That is, the density function of T;, is the 
convolution product of the density functions of 7),... ,7n—1- 


In some cases, we know the exact distribution of the random variable T,,. 
For example, if 7; ~ Exp(A), then 


Tr ~ G(n,A) (5.156) 
Similarly, if 7; ~ Poi(A), we have 

Tr ~ Poi(nd) (5.157) 
If n is large enough, we deduce from the central limit theorem that 

n © N(np, no”) (5.158) 
where p := E[7;] and o? := V[7] Vi. 


Example 5.6.1. We can use the formula (5.153) to check that, for a Poisson 
process with rate A, we have 


PIN(t) =n] =e AP fort > 0 andn=0,1,... 


In this case, 


t 
Tr ~G(n,\) => Pit. <= [ hers? 
0 


Let : 
Tn =] eg" dr forn =0,1,... 
0 
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Integrating by parts, we obtain 


e7 n 
I, = —£"—— ~In- 
a Ma xf 1 
a n} nena e n! 
ao fee —I 
2 akeI" F co 
= 23 nt pn—kt1 © a nl I 
~ £(n=k+0) a as 
where : tes 
l1-—e7 
_ —Ax = 
Io -/ e€ dx > 
It follows that 
ACT - n! et ont (1-e-*# 
Yr ee - gn— kt r 
Ener SE = | & (n—k+1)! Ne Qn r 
. LG Atnokts us (At)? x 
a es Ma ee 
me i! 
ae (n—k+1)! aor a 
Similarly, 
n-1 : 
= (At) at 
Pins ts ye a 


Thus, we indeed have 


PIN() =n) = ( 2 3 ore) 7 ( SS aoe) = ON oan 


i=0 i=0 


When 7; is a discrete random variable, the probability P[r; = 0] may be 
strictly positive. That is, the time needed for a renewal to occur may be equal 
to zero. For example, if 7; ~ Poi(A), then P[7; = 0] = e~* > 0. However, if 
the nonnegative random variable 7; is not the constant 0, we may write that 
Lt > 0. If we assume that pp < 00 and a? < 0, the strong law of large numbers 
(see p. 32) then implies that 


n—-1 
lim a Ti = 
n—-0o n 


1=P 


from which we deduce from the formula (5.151) that we can write 


P[N(t) = 00] =0 for all t< 00 (5.160) 
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because the random variable T,, will eventually be larger than any finite ¢. 
That is, there cannot be an infinite number of renewals in a finite-time interval. 
However, given that P[r; = oo] = 0, we may conclude that 


P [ jim, N(t) = 00] =1 (5.161) 


Since, in most cases, it is very difficult to explicitly calculate the probability 
of the event {N(t) =n}, we must generally content ourselves with finding the 
mean of the random variable N(t). 


Definition 5.6.2. The function my(t) := E[N(t)] is called the renewal 
function (or mean-value function) of the renewal process {N(t),t > O}. 
Proposition 5.6.2. The renewal function mn (t) can be calculated as follows: 


mn(t) = ‘s PIT, <t] (5.162) 


n=1 


Proof. As N(t) is a random variable taking its values in the set N° = 
{0,1,...}, we may write (using the relation T, <t <= N(t) > n) that 


EIN(t)] = S\iPIN(t) = i] = SCiPIN() =) =S0 0 PIN® = 7] 
1=0 i=1 i=l n=1 
= VD PIN() =i] =D PIN 2] 
= 3 PR st) (5.163) 


Remarks. i) Since it is also difficult to calculate the probability P[T,, < t], in 
practice the proposition does not enable us very often to obtain the function 
mn (t). Moreover, if we managed to calculate P[N(t) = n], for all n, then it 
is perhaps simpler to find the mean of N(t) by directly using the definition of 
E[N(#)].- 

ii) We will further discuss another technique that enables us, when the 7;’s 
are continuous random variables, to calculate my(t), namely by solving an 
integral equation. 


Example 5.6.2. Suppose that 7; has a Bernoulli distribution, with parameter 
p € (0,1). We then have 


n—-l1 
Te e= S 7%; ~ B(n,p) 
i=0 
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so that ‘a 
t 
n\ k n—-k 
PIT, <t) = 1- 
| | » (je (1-p) 


where [{t] denotes the integer part of t. Theoretically, we can calculate my (t) 
by finding the value of the sum 


where ) =Oifk>n. 


Consider the particular case when p = 1/2. We have 


S(t,1/2) = . fapry ei, 


n=1 k=0 


We find, making use of a mathematical software package, that S(0,1/2) =1, 
S(1,1/2) = 3, $(2,1/2) = 5, $(3,1/2) = 7, etc., from which we deduce the 
formula 

S(t, 1/2) = 2|t]+1 for allt >0 


Now, let t =r € {0,1,...}. Given that 7; = 0 or 1, for all 7, we may write 
that N(r) > r. We find that 


r with probability p’t! 


1 
r + 1 with probability (" \orna —p) 
1 
N(r) = 


2 
r +2 with probability (" i ) pti(1 —p)? 


In general, we have 
PIN(r)=r+kl= (CE erta- 2 fork =0,1,... 


Remark. We have that P{N(r) = r] = p’t!, and not p", because {N(r) =r} if 
and only if (iff) the first r + 1 renewals each take one time unit. In particular, 
N(0) = 0 iff 79 = 1. Similarly, N(1) = 1 iff 7 =1 and 7 = 1, ete. 

We may write that X := N(r) +1 has a negative binomial distribution 
(see p. 135) with parameters r+ 1 and p. Now, the mean of this distribution 
is given by (r + 1)/p, from which we deduce that 
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BINS 1 = pweys 


P P 


Note that if p = 1/2, then we retrieve the formula S(t, 1/2) = 2[t]+1 obtained 
previously. 


Suppose that the 7;’s are continuous random variables. We then deduce 
from Eq. (5.162) that 


<mn() => fr,(t) (5.164) 
n=1 


from which, taking the Laplace transform of both members of the preceding 
equation, we obtain 


m(a) = [ ” etdmu(t) = I ” eet fp (t) dt (5.165) 
n=l 
— my(a)= s Mr, (a) (5.166) 
n=1 


where Mr, is the moment-generating function of the random variable T,, 
(see p. 19) and a is a negative constant. Since the 7;’s are independent and 
identically distributed random variables, we may write that 


n-1 
Mr, (a) "S" T] M,,(a) = [M,,(a)]” (5.167) 
= 


The parameter a being negative, we have that M,,(a) € (0,1). It follows that 


mi(a) = J Mn (a)]" = tr (5.168) 
so that 
= my (a@) 
M,, (a) = T+ me(a) (5.169) 


By the uniqueness of the Laplace transform (and of the inverse Laplace trans- 
form), we deduce from the last equation that to each renewal function my (t) 
corresponds a different density function f,,(t). Similarly, Eq. (5.168) implies 
that the density function f,,(f) uniquely determines the function my(t). 
Moreover, it can be shown that these results hold true, whether the ran- 
dom variables 7; are discrete or continuous. We can thus state the following 
proposition. 
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Proposition 5.6.3. There is a one-to-one correspondence between the distri- 
bution function of the random variables 7; and the renewal function mn (t). 


Since my (t) = At in the case of the Poisson process, we have the following 
corollary. 


Corollary 5.6.1. The Poisson process is the only renewal process having a 
linear renewal function. 


Remark. The Poisson process is also the only Markovian renewal process. 


In the case when the random variables 7; are continuous, we have 


my(t) := B[N(t)] = E[E[N(t) | ro]] = [awe | 70 = T|fro (7) dr 


(5.170) 
Moreover, the relation 
TM™>t <— MN(t)=0 (5.171) 
implies that 
t 
my (t) -{ E(N(t) | 70 = tT] fr (7) ar (5.172) 
0 


Finally, we may write (the 7,’s being i.i.d. random variables) that 
E(N(t) | 7 =7] =1+E(N(t-—7)]=1+mn(t-—7) forO<7<t (5.173) 
It follows that 
t 
mn (t) = Mi [1+ mn(t —7)] fro (7) dr (5.174) 
0 
t 
eS iy Oe Ph 4 [ NEA PEde wei: Wa) 
0 


Definition 5.6.3. Equation (5.175) is called the renewal equation (for the 
renewal function) of the process {N(t),t > 0}. 


To explicitly calculate the renewal function, it is often easier to solve the 
integral equation (5.175). Setting s = ¢ — 7, we can rewrite it as follows: 


mn (t) = Fy, ( y+ ff mn(s)f7,(t-—s) ds fort >0 (5.176) 


Next, differentiating both members of the preceding equation, we obtain 


mly(t) = frp(t) + mu (t) frp (0 +f my(s)fi,(t—s) ds (5.177) 
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When f,,(t — s) is a polynomial function, we can differentiate repeatedly 
the members of this equation, until we obtain a differential equation for the 
function my (t). We have, in particular, the boundary condition my (0) = 0. 


Remark. We can also sometimes obtain a differential equation for my (t) even 
if f)(t — s) is not a polynomial function. 


Example 5.6.3. Let 
fro(t) = 408 if0<t<1 


We have [see Eq. (5.177)]: 
t 
my (t) = 42 + my (t) x 0 +f my (s)12(t — s)? ds 
0 
t 
=A +f my(s)12(t—s)? ds forO<t<1 
0 
Note that this equation implies that m‘,(0) = 0. We differentiate once more: 
t 
mit, (t) = 120? +/ my (s)24(t — s) ds 
0 
from which we deduce that m‘,(0) = 0. Next, we have 
t 
my (t) = 24t + a | mn (s) ds 
0 
so that m/x;(0) = 0. Finally, we obtain the ordinary differential equation 
mY (t) = 24+ 24my(t) 
n (tb) = N 


The general solution of this equation is given by 


mn (t) = -1+ 1 cos kt + co sin kt + cge*t + cge* 


where k := (24)!/4 and cy, ..., c4 are constants. The particular solution that 
satisfies the conditions my (0) = miy(0) = m4,(0) = my (0) = 0 is 


1 1 
my (t) = -1+ 3 cos kt + 7 (ek + ec) forO<t<1 
Note that the solution above is only valid for t € [0,1]. Moreover, since the 
random variable 7) takes its values in the interval [0, 1], the mean my (1) must 


be greater than 1. We calculate 


1 1 
my (1) =—1+ 5 cosk +3 (e* +e7*) ~ 1.014 
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Equation (5.175) is a particular case of the general renewal equation 
CO 
g(t) = h(t) + [ g(t-r)dF,,(7) fort >0 (5.178) 
0 


where h(t) and F;,,(¢) are functions defined for all t > 0. The following propo- 
sition can be proved. 


Proposition 5.6.4. The function g(t) defined by 
t 
g(t) = A(t) + h(t-—1) dmy(r) fort >0 (5.179) 
0 
is a solution of the renewal equation (5.178). 
Remarks. i) Equation (5.175) is obtained from (5.178) by taking g(t) = mn(E) 
and h(t) = F,,(t) and by noticing that my(t) = 0 if t < 0. If A(t) is a function 
bounded on finite intervals, then g(t) is bounded on finite intervals as well. 
Moreover, g(t) is the unique solution of (5.179) with this property. Note that 


the distribution function F,,(t) (= h(t)) is evidently a bounded function, 
because 0 < F,,(t) <1 Vt>0. 


ii) We can replace dmy(r) by my(r) dr in the integral above. 


Corollary 5.6.2. If 79 is a continuous random variable, then the second mo- 
ment of N(t) about the origin, E[N?(t)], is given by 


E[N?(t)| = mn(t) + 2 f myn(t—7T)dmy(t) fort >0 (5.180) 
0 
Proof. We may write that 
E(N?(t)] = | 7 E[N?(t) | to = 7] fro(7) dr (5.181) 
t t 
= f BNW) |= Tifa(r) ar = f BA +N - 7) fool) ar 
0 0 
= {1 + 2E[N(t —7)] + E[N7(t — T)}} fro(7) dr 
: t t 
= F,,(t) + 2 | mn(t —T)fry(7) ar +f E[N?(t —7)] fro (7) ar 
0 0 
Making use of Eq. (5.175), we obtain 


BIN*()] = 2m (t) — Felt) + fo BINED Mfa(7) dr (5.182) 
0 
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This equation is of the form of that in (5.178), with g(t) = E[N*(t)] and 
h(t) = 2my(t) — F;,,(t) (since g(t) = 0 if t < 0). It follows, by Proposition 
5.6.4, that 
t 
E(N?(t)] = 2my(t) — F,,(t) +f [2my(t — 7) — F,,(t — 7)] dmn(r) 
0 
(5.183) 
We have 


i 


| F,,(t —7) dmy(t) F,,,(t — r)mn(T)I6 + i fro(t —T)mn (7) dr 
0 0 


\| 


t 
0 +/ fro(T)mn(t — 7) dr 
0 
O27) my(t) — F(t) (5.184) 
from which we obtain Eq. (5.180). 0 
Example 5.6.4. When {N(t),t > 0} is a Poisson process with rate A, we have 
E(N?(t)] = VIN(t)] + (EIN(#)])? = At + At)? 


which is indeed equal to 


my (t) + 2f my(t —7) dmy(r) = At+ 2f A(t — r)A dr 


On re 
= At +2? (P-5) 


fro(t)=1 if0<t<1 


Example 5.6.5. When 


we find that 
E(N(t)}=e'—1 for0<t<1 


It follows that 
t 
E[N?(t)] = e& —1+ 2 | (e-7 —1)e” dr 
0 


t 

=e'-142/ (e' —e”) dr =e' ~1+2[te’ — (e* —1)] 
0 

=e'(2t-1)+1 for0<t<1 


As in Example 5.6.3, the fact that 0 < 7) <1 implies that E[N*(1)] > 1, 
for k = 1,2,.... We have 


E[N(1)]=e-1 and E[N?(1)}=e+1 
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We know that the sum of two independent Poisson processes, with rates A; 
and Ag, is a Poisson process with rate \ := A1 + Az (see p. 234). The following 
proposition can be proved. 


Proposition 5.6.5. Let {Ni(t),t > 0} and {No(t),t > 0} be independent 
renewal processes. The process {N(t),t > 0} defined by 


N(t) = Ni(t)+No(t) fort >0 (5.185) 


is a renewal process if and only if {N;(t),t > 0} is a Poisson process, for 
=, 2. 


Finally, the following result can be proved as well. 


Proposition 5.6.6. Let {N(t),t > 0} be a renewal process. The kth moment 
of N(t) about the origin (exists and) is finite, for all k € {1,2,3,...}: 


E(N*(t)] < 00 for allt < 00 (5.186) 


Remark. We deduce from the proposition that my (t) < oo Vt < oo. Thus, not 
only can the number of renewals in an interval of finite length not be infinite 
(see p. 270), but the mathematical expectation of the number of renewals in 
this interval cannot be infinite either. 


5.6.1 Limit theorems 


Note first that Ty(z) designates the time instant of the last renewal that 
occurred before or at time t. Similarly, Ty()41 denotes the time instant of 
the first renewal after time t. 

We will show that the average number of renewals per time unit, namely 
N(t)/t, tends to 1/u as t — oo, where pw := E[7;] V 7 is assumed to be finite. 
The constant \ := 1/p is called the rate of the process. 


Proposition 5.6.7. We have, with probability 1: 


MO, ast +00 (5.187) 
where X = 1/pu > 0. 
Proof. We may write that 
Tyty St < Tye)41 (5.188) 
NO ests INDE (if N(t) > 0) (5.189) 


N(t) ~ N(t) NZ) 


Now, by the strong law of large numbers (since E]|7;|] = E{7;] < 00), we have 
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N(t)-1 


T. 
ne Pie PG +p as N(t) > 00 (5.190) 


with probability 1. Furthermore, because N(t) — oo as t > oo, we may write 
that (with probability 1) 


im ne = a (5.191) 
Finally, given that 
Pyejti _— Tnie+1 (1+ x) (5.192) 
N(t) N(t)+1 N(t) 
we deduce that 
@) 1 


p< lim —— WO <p = (5.193) 


too N(t 


with probability 1.0 


The next theorem, which states that the average expected number of re- 
newals per time unit also converges to A, is not a direct consequence of the pre- 
ceding proposition, because the fact that a sequence De aa of random vari- 
ables converges to a constant c does not imply that the sequence {E|X,]}?__, 
converges to c. 


Theorem 5.6.1 (Elementary renewal theorem). Jf the mean Ejr0] is fi- 
nite, then we have 
van EAN 


t—co 


=i (5.194) 


Remark. i) Actually, if 2 = oo, the preceding result is still valid (by setting 
A := 1/p = 0). Similarly, Proposition 5.6.7 is valid when jz = oo as well. 

ii) There is no mention of probability in the theorem, because E[N(t)] is a 
deterministic function of t (while N(t) is a random variable). 


The following two theorems can also be shown. 
Theorem 5.6.2. If := Elo] < 00 and o? := V[r0] < 00, then 
V 
fn VINCI 


too 


=o7}3 (5.195) 


Theorem 5.6.3 (Central limit theorem for renewal processes). If 
E{ré] is finite, then we may write that 


2 
N(t)=N (£. =) =N (At,o*\*t) if t is large enough (5.196) 
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Example 5.6.6. If {N(t),t > 0} is a Poisson process with rate Ao, then 7 ~ 
Exp(Ao), so that yo := E[79] = 1/Xo and o? := V[79] = 1/Ag. We have indeed 


N(t) ~ Poi(Aot) & N(Aot, Aot) = N(At, Ap 2A*t) 
where A := 1/p = Ao. 


The proposition that follows can be used to show Theorem 5.6.1. 
Proposition 5.6.8. We have 
E(Tw +1] = Elo] {E[N (t)] + 1} (5.197) 


Proof. By conditioning on the value taken by the random variable 79, which 
is assumed to be continuous, we obtain 


E{Tw(t)+1] = | E{Tyt)41 | 70 = T]fro(7) ar 
t love) 
= [fet Bite nsil} hole) art [ rhe(r) ar 
0 t 
foe) t 
= i) T fro(T) dr + E[Tn er) 41) fro (7) dr 
0 0 


= El] + [ ElTyte-rysilfro(7) ar (5.198) 


This equation is the particular case of the renewal equation (5.178), which is 
obtained with g(t) = E[Ty)41] and h(t) = E[ro]. Dividing both members of 
the equation by E[79] and subtracting the constant 1, we find that 
_ Elfy@ii) | _ [ E(Twe-7)+1] 
0 


g(t) = ————- -1= Ero] 


Etro] fro(T) at (5.199) 


t 


t 
oF | [g"(t—7) +1] fro(7) dt = Fro(t) + i g"(t — 7) fro(7) dr 
0 0 


Now, this equation is the same as Eq. (5.175). Therefore, by the uniqueness 
of the solution, we may write that 


ET ¢)+1] 


MmN (t) = E\ro] 


-1 = Elfyqy4i] = Efrollmn(t)+1] 9 
(5.200) 


Definition 5.6.4. Let t be a fized time instant. The random variable 


A(t) = t — Ty (5.201) 
is called the age of the renewal process at time t, while 


is the remaining or excess lifetime of the process at time t (see Fig. 5.3). 
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Fig. 5.3. Age and remaining lifetime of a renewal process. 


By proceeding as above, we can prove the following proposition. 


Proposition 5.6.9. The distribution function of D(t) is given by 


t 
Fow(t)=Fatt+r)- [l= Falt+r—r)] dmy(r) (6.203) 
0 
forr > 0. In the case of A(t), we find that 


P[A(t) > a] =1- F,,(t) + fr —F,,(t-7)] dmn(r) if0<a<t 
: (5.204) 
and P[A(t) > al =0 ifa>t. 


When ¢ is large enough and 79 is a continuous random variable, we find 
that 
1 
fow(r) = Emm" —F,,(r)] forr > 0 (5.205) 
In fact, this formula for the probability density function of D(t) is exact if the 
renewal process {N(t),t > 0} is stationary. 

Suppose now that at the moment of the nth renewal of the process 
{N(t),t > 0}, we receive, or we have accumulated, a reward R, (which may, 
actually, be a cost). Suppose also that the rewards {R,,}>-, are independent 
and identically distributed random variables. However, in general, Ry will de- 
pend on 7,1, that is, the length of the nth renewal period, called a cycle. Let 


R(t) be the total reward received in the interval [0,t]. That is, 


N(t) 
R(t)= 5° Rn (R(t) = 0 if N(t) = 0) (5.206) 


We will show that the average reward received by time unit, in the limit, is 
equal to the average reward received during a cycle, divided by the average 
length of a cycle. 


Proposition 5.6.10. If E{R,| < oo and E[ro] < 00, then we have 


_ R(t) E[Rij] _ 
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Proof. It suffices to use the strong law of large numbers and Proposition 
5.6.7. Since N(t) — 00 as t — 00, we may write 


NW) p 
P | jim > MH 7 E(Ry}| =1 (5.208) 


from which we have 


too ot too | 4 N(t 
1 
E({R 5.209 
[Ri] x Elro] ( ) 
with probability 1. 0 
Remarks. i) It can also be shown that 
_ E[R(t)] _ Eli] 5 

=— .210 
jin | Elro| (5.210) 


ii) If the reward R, may take positive and negative values, then we must 
assume that E||R,|] is finite. 


With the help of the preceding proposition, we will prove the following 
result. 


Proposition 5.6.11. If 7 is a continuous random variable, then we have, 
with probability 1: 
fo D(r) dr _ Elrd| 


. So A(1 ) di 5 
jim t jim t 9 E| i ( ) 


Proof. Suppose that the reward received at, time t is given by A(t). Then, we 
may write that 


E[Ri\|=E iE ; ar| = 2/2] 2 Firs] (5.212) 


To obtain the other result, we simply have to suppose that the reward received 
at time ¢ is rather given by D(t), and then the variable 7 is replaced by 7 — 7 
in the integral. 0 

Remark. Note that we calculated the average value of the age (or of the re- 
maining lifetime) of the process over a long period, which is a temporal mean, 
and not a mathematical expectation. If E[73] < oo, then it can also be shown 
that. 


E{ré] 
2E|70| 


lim E[A(t)] = jim E[D(t)| = (5.213) 


t-+00 
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Example 5.6.7. If {N(t),¢ > 0} is a Poisson process, then, making use of the 
fact that it has independent and stationary increments, we can assert that 
D(t) has an exponential distribution with parameter A. Thus, we have 

1 


EID(t)| = 5 


Moreover, 7 ~ Exp(A), too, so that 


a 2 
ina So D(x) dr _ xz +() i 
t00 t 2/ a 
So, in this case, the temporal mean and the mathematical expectation of D(t) 
are equal, for all t > 0. 


Example 5.6.8. (a) Let {N(t),t > 0} be a counting process for which the time 
between two events has a U/0, S] distribution, where 


g — J 1 with probability 1 /2 
~ | 2 with probability 1/2 


Explain why {N(t),t > 0} is not a renewal process. 
(b) In part (a), suppose that S$ is rather a random variable whose value is 
determined at time t = 0 and after each event by tossing a fair coin. More 


precisely, we have 
$= 1 if “tails” is obtained 
~ | 2 if “heads” is obtained 


In this case, the stochastic process {N(t),t > 0} is a renewal process. 

(i) Calculate my(t), forO<t <1. 

(ii) Suppose that we receive a reward equal to $1 when the length of a 
cycle is greater than 1 (and $0 otherwise). Calculate the average reward per 
time unit over a long period. 


Solution. (a) {N(t),t > 0} is not a renewal process, because the times be- 
tween the successive events are not independent random variables. Indeed, 
the smaller 7 is, the larger the probability that S = 1 is. So, the 7,’s depend 
on 79, for all k > 1. 


(b) (i) Let 7 be the length of a cycle. We have 
F(t) = Pir <t)= Pir <t|S =1)P[S =1])+P[r <t| S=2)P[S =2] 
a ee 
1/2 
That is, 


2 


It follows that the renewal equation is 


1 t 
Fi) =5 (#45) =4 for0<t<1 
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3t ; Bs weice Sh ONT" 
th=— ay sg ee ee g 
my (t) 1 4 : mit «)7 d 1 4 q mn (y) dy 
3 
U a 
=> my(t) at qin (t) 


We deduce from the initial condition my (0) = 0 and from the formula (3.233), 
p. 130, that 


t 
my (t) = e¢/4 (0+ f ewes ay) =e*/4_1 for0<t<1 
0 
(ii) We have 
3 


Bln] = Blry| $= 1] PIS =I] +l | S$ =2)P(5 =2)= 3 (5 +1) =4 


1/2 


and 


E[Ri] = E{Ry | $= 1] P[S = 1] + ElRi | 9 =2) PS = 2) = 5 


0 1/2 


Therefore, the average reward per time unit (over a long period) is given by 


(1/4)/(3/4) = 1/3. 


5.6.2 Regenerative processes 


We know that the Poisson process, {N(t),t > 0}, starts anew, probabilisti- 
cally, from any time instant, because it has independent and stationary incre- 
ments. However, since it is a counting process, once the first event occurred, 
N(t) will never be equal to 0 again. We are interested, in this subsection, 
in processes that are certain to eventually return to their initial state and 
that, at the moment of this return, start afresh probabilistically. This type of 
stochastic process is said to be regenerative. 


Definition 5.6.5. Let tT) be the time that the discrete-state stochastic process 
{X(t),¢ > 0} spends in the initial state X(0). Suppose that 

i) PIA t > 7: X(t) = X(0)] =1, 

ii) the processes {X(t) — X(0),t > O} and {Y(t),t > O}, where 


Y(t) = X(t +71) — X(T) (5.214) 


and T, is the time of first return to the initial state, are identically distributed, 
iit) the stochastic process {Y(t),t > 0} is independent of the process {X(t), 
0<t<Tj}. 

The process {X(t),t > 0} is then called a regenerative process. 
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Remarks. i) Let T, be the time of the nth return to the initial state, for 
n =1,2,.... We assume that P/T < oo] = 1. It then follows that the time 
instants 7>,73,... must also (exist and) be finite with probability 1. 

ii) A regenerative process may be a discrete-time process {Xn,n = 0,1,...} 
as well. In this case, 79 is the time that the process takes to make a transition 
from the initial state to an arbitrary state, which may be the initial state (for 
example, if {X,,n =0,1,...} is a Markov chain). 


iii) The cycles of the regenerative process are the processes 


{Co(t),0<t< Ty} and {Cy(t),Tn <t<Tayi} forn=1,2,... 


(5.215) 
These stochastic processes are independent and identically distributed, for 
n=0,1,.... Similarly, the random variables Xo := T; and Xp := Tn4i — Th, 
forn =1,2,..., are i.i.d. 


iv) According to the definition above, a renewal process is not regenerative, 
because it is a particular counting process. However, we can associate a re- 
newal process {N(t),t > 0} with a regenerative process {X(t),t > 0}, by 
setting that N(t) is the number of times that {X(t),t > 0} came back to 
the initial state and started anew in the interval [0,t]. That is, N(t) is the 
number of cycles completed in this interval. Conversely, we can define a re- 
generative process {X(t),t > 0} from a renewal process {N(t),t > 0} (for 
which N(0) = 0) as follows: 


Nit) if0<t<T? 
_ | NQ)—k fT? <t <TR 
X(Q°= 9 w(t) — 2k if Te, <t < Th C7) 


for some k € {2,3,...}, where 7}, 73,... are the arrival times of the renewals. 
Note that with k = 1, we would have that X(t) = 0. Moreover, here 7; = Tf. 


Example 5.6.9. The queueing model M/A1/1, described in Example 3.1.6 (and 
which will be studied in detail in Chapter 6), is an example of a regenerative 
process. In this model, X(t) designates the number of persons in a system at 
time ¢. Thus, if X(0) = 0, then we are certain that the system will eventually 
be empty again and that, from that point on, everything will start anew. 


Example 5.6.10. Let {Xn,n = 0,1,...} be a discrete-time Markov chain, 
whose state space is the set {0,1} and whose one-step transition probabil- 


ity matrix is given by 
— {1/2 1/2 
gee 


Suppose that Xo = 0. Since the chain is irreducible and has a finite number 
of states, we may assert that it is recurrent. Therefore, the probability that 
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the process will visit the initial state 0 again is equal to 1. It follows that this 
Markov chain is a regenerative process. Suppose that every transition of the 
chain takes one time unit. Then, we may write that P[T; = 1] = P[T; = 2] = 
1/2. Finally, let N(0) =0 and N(t) be the number of visits to state 0 in the 
interval (0,t], for t > 0. The process {N(t),¢ > 0} is then a renewal process. 


Proposition 5.6.12. If E[T;| < oo, then the proportion 7, of time that a 
regenerative process spends in state k is given, in the limit, by 


E E|time spent in state k in the interval [0, T,]] 
b=-eroOCOr..e.—cc ee 


A] (5.217) 


Proof. Suppose that we receive an instantaneous reward of $1 per time unit 
when the process {X(t),t > 0} is in state k, and $0 otherwise. Then the total 
reward received in the interval [0,¢] is given by 


t 
R(t) =| Ty x(r)=k} aT (5.218) 
0 
where I;x(7)=x} is the indicator variable of the event {X(T) = k} [see 
Eq. (3.43). If E[T1] < oo, Proposition 5.6.10 implies that 
AD EE (5.219) 


Fareel t = E{T\] 


(with probability 1), where R, is the time spent by the process in state k 
in the interval [0,71]. Now, limz... R(t)/t is the proportion of time that the 
process spends in state k. 0 

Remark. The proposition is valid for any type of random variable T;. When 
T, is a continuous variable, it can also be shown that 


jim P[X(t) =k] =m (5.220) 


That is, 7, is the limiting probability that the process will be in state k at 
time t as well. 


Example 5.6.11.In the case of the process {X(t),t > 0} defined by Eq. 
(5.216), we can directly write that m9 = 71; =... = 7-1 = 1/k, because 
the random variables Ty, Tj, ... are independent and identically distributed. 


Example 5.6.12. In Example 5.6.10, the reward R, is equal to 1, because the 
Markov chain spends exactly one time unit in the initial state 0 per cycle. It 


follows that 
1 1 2 


—~ET| 3/23 

We can check, using Theorem 3.2.1, that the proportion 7 is indeed equal 
to 2/3. Note that we may apply this theorem to obtain the proportion 709, 
whether the chain is periodic or not. Here, the chain is ergodic, since po,o = 
1/2 > 0. Then, 70 is also the limiting probability that the process will be in 
state 0 after a very large number of transitions. 


70 
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Example 5.6.13. We consider a discrete-time Markov chain {X,,n =0,1,...} 
with state space {0,1,2} and transition matrix P given by 


01/21/2 
P=|10 0 
01/21/2 


Suppose that the initial state is 0 and that when the process enters state 7, it 
remains there during a random time having mean ju;, for 7 = 0,1, 2, indepen- 
dently from one visit to another. Calculate the proportion of time that the 
process spends in state 0 over a long period. 


Solution. The Markov chain is recurrent, because it is irreducible and has a 
finite number of states. Then the process {X;,,n = 0,1,...} is regenerative. 
Let T; be the time of first return to state 0 and N be the number of transitions 
needed to return to 0. We have 


E(t, | N =n) =potpit(n—2)u2 forn=2,3,... 


Since P[N =n] = (1/2)"~!, for n = 2,3,..., it follows that 


E(Ty] = E(E(T; | N]] = So [uo + pa + (n — 2)uo)(1/2)" 


n=2 
Finally, we can write that 
oO CO 
do (n= 2)(1/2)""4 = So (n — 1)(1/2)" = E[Geom(1/2)] - 1 =1 
n=2 n=1 
so that 


E(T\] = wo + bt + be 


The proportion of time that the process spends in state 0 over a long period 
is thus given by uo/(uo + fa + p2). 

Remark. Since the matrix P is finite and doubly stochastic, we deduce from 
Proposition 3.2.6 (the Markov chain being irreducible and aperiodic) that the 
limiting probabilities 7, exist and are given by 1/3, for k = 0,1,2. Hence, we 
deduce that the proportion of time requested is indeed equal to uo/(“o + ui + 
2). 


An important particular case of a regenerative process is the one for which 
the state space of the process { X(t),t > 0} contains only two elements, which 
will be denoted by 0 and 1. For example, X(t) = 0 if some machine is down, 
and X(t) = 1 if it is operating at time t. Suppose that the machine is brand- 
new at the initial time 0, so that X(0) = 1, and that the time during which 
the machine operates, before breaking down, is a random variable S$). Next, 
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the repairman takes a random time U, to set the machine going again, and 
then the system starts afresh. Let 


Ty := S,+U, forn=1,2,... (5.221) 


where S,, is the operating time of the machine between the (n — 1)st and 
the nth breakdown, and U,, is the repair time of the nth breakdown, for 
n = 2,3,.... We assume that the random variables {S,,}?-., are iid. and 
that so are the r.v.s {U, }°-_,. However, the r.v.s S, and U, may be dependent. 
This type of stochastic process is often called an alternating renewal process. 


Remark. Let N(t) be the number of times that the machine has been repaired 
in the interval [0,t]. The process {N(t),t > 0} is a renewal process and the 
T,’s are the arrival times of the renewals. However, the process {X (t),t > 0} 
defined above is not a renewal process (according to our definition), because 
it is not a counting process. 


By using the renewal equation, we can show the following proposition. 
Proposition 5.6.13. Let 7, (t) be the probability that X(t) =1. We have 
t 
PAC ee es | lt — Fe,(¢—7)] dmy(r) (5.222) 
0 


Next, we deduce directly from Proposition 5.6.12 that 
EISi) EIS] 


i eh Se 5.223 
"= FS) + 20 E(t] eae 
Moreover, if ZT) is a continuous random variable, then 
i = 224 
Jim m(t) =m (5.224) 
Finally, let {N(t),t > 0} be a renewal process and 
__ J lif A(t) <a 7 
X(t) := ‘ if A(t) >a (5.225) 


where a > 0 is a constant and A(t) is defined in Eq. (5.201). The process 
{X(t),t > 0} is then an alternating renewal process, and we deduce from the 
formula (5.223) that 

_ E{S\} — Elmin{Ty, a}] 


where Ty‘ is the arrival time of the first renewal of the stochastic process 
{N(t),t 2 O}. 
Remark. lf TY < a, then we set U, = 0. 
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In the continuous case, we have 


E{min{Ty,a}] = We min{t,a}frs(t)dt = [ tfr;(t)dt + a afrrs(t)dt 
(5.227) 


We calculate 
a a a a 
/ t fr: (t)dt = tks (t)|, = ‘i Fr- (t)dt = aFr- (a) = i: Fr; (t)dt (5.228) 
0 0 
from which we obtain the formula 


Elmin{Tj , a}] = aFy;(a) — is Fr; (t)dt + aP|TY > al = a— Ve Frs (t)dt 
(5.229) 


We can thus write that 


Jo PIT? > t]dt 


Ty (5.230) 


VS 


Remark. We obtain exactly the same formula as above for 7 if we replace 
A(t) by D(é) (defined in (5.202)) in (5.225). 


Example 5.6.14. If Ts ~ Exp(A), then we have that P[Tf > t] = e~**. It 


follows that a 
_ AE e7** dt 7 


i —dra 
i/X : 


us| 
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Section 5.1 


Question no. 1 
Let {N(t),t > 0} be a Poisson process with rate A. We define the stochastic 
process {X(t),0 <t < c} by 


X(t) = N(t)- ~N(o) for 0<t<c 


where c > 0 is a constant. 

(a) Calculate the mean of X(t). 

(b) Calculate the autocovariance function of the process {X(t),0 <t < c}. 
(c) Is the process {X(t),0 < t < c} wide-sense stationary? Justify. 
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Question no. 2 

We suppose that customers arrive at a bank counter according to a Poisson 
process with rate 4 = 10 per hour. Let N(t) be the number of customers in 
the interval [0, t]. 
(a) What is the probability that no customers arrive over a 15-minute time 
period? 
(b) Knowing that eight customers arrive during a given hour, what is the 
probability that at most two customers will arrive over the following hour? 
(c) Given that a customer arrived during a certain 15-minute time period, 
what is the probability that he arrived during the first 5 minutes of the period 
considered? 


(d) Let X(t) := N(t)/t, for t > 0. Calculate the autocovariance function, 
Cx (t1,t2), of the stochastic process {X(t),t > O}, for t1,t2 > 0. 


Question no. 3 
The stochastic process {X(t),t > 0} is defined by 
_ N(t+0?) — N(t) 


X(t) == for t 2 0 


where {N(t),t > 0} is a Poisson process with rate 4 > 0, and 6 > Oisa 
constant. 
(a) Is the process {X(t),t > 0} a Poisson process? Justify. 
(b) Calculate the mean of X(t). 
(c) Calculate the autocovariance function of the process {X(t),t > 0}, for 
ty =1, to =2, andé=1. 
(d) Let Z, := N(n), forn =0,1,2,.... 

(i) The stochastic process {Z,,n = 0,1,...} is a Markov chain. Justify 
this assertion. 

(ii) Calculate p;j, for i,j € {0,1,2,...}. 


Question no. 4 

Let N(t) be the number of failures of a computer system in the interval 
(0, t]. We suppose that {N(t),t > 0} is a Poisson process with rate 4 = 1 per 
week. 


(a) Calculate the probability that 
(i) the system operates without failure during two consecutive weeks, 


(ii) the system will have exactly two failures during a given week, knowing 
that it operated without failure during the previous two weeks, 


(iii) less than two weeks elapse before the third failure occurs. 
(b) Let 


Z(t):=eN fort >0 
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Is the stochastic process {Z(t),¢t > 0} wide-sense stationary? Justify. 
Indication. We have that E[e~**] = exp [a (e~* — 1)] if X ~ Poi(a). 


Question no. 5 

A man plays independent repetitions of the following game: at each repe- 
tition, he throws a dart onto a circular target. Suppose that the distance D 
between the impact point of the dart and the center of the target has a U[0, 30] 
distribution. If D < 5, the player wins $1; if 5 < D < 25, the player neither 
wins nor loses anything; if D > 25, the player loses $1. The player’s initial 
fortune is equal to $1, and he will stop playing when either he is ruined or his 
fortune reaches $3. Let X, be the fortune of the player after n repetitions. 
Then the stochastic process {X,,n = 0,1,...} is a Markov chain. 


(a) Find the one-step transition probability matrix of the chain. 
(b) Calculate E[X3]. 


Suppose now that the man never stops playing, so that the state space of 
the Markov chain is the set {0,+1,+2,...}. Suppose also that the duration T 
(in seconds) of a repetition of the game has an exponential distribution with 
mean 30. Then the stochastic process {N(t),t > 0}, where N(t) denotes the 
number of repetitions completed in the interval [0, t], is a Poisson process with 
rate A = 2 per minute. 


(c) Calculate the probability that the player will have completed at least three 
repetitions in less than two minutes (from the initial time). 


(d) Calculate (approximately) the probability P{[N(25) < 50]. 


Question no. 6 

Let N(t) be the number of telephone calls received at an exchange in the 
interval [0,t]. We suppose that {N(t),t > 0} is a Poisson process with rate A 
= 10 per hour. Calculate the probability that no calls will be received during 
each of two consecutive 15-minute periods. 


Question no. 7 

The customers of a newspaper salesperson arrive according to a Poisson 
process with rate A = 2 per minute. Calculate the probability that at least 
one customer will arrive in the interval (to, to + 2], given that there has been 
exactly one customer in the interval (tp ~ 1, to +1], where tp > 1. 


Question no. 8 
The stochastic process {X(t),t > 0} is defined by 


X(t)=N(t+1)-N(1) fort >0 


where {N(t),¢ > 0} is a Poisson process with rate \ > 0. Calculate C'x(s, t), 
forO<s<t. 
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Question no. 9 We define 


where {N(t),t > 0} is a Poisson process with rate A = 1. It can be shown that 
P{Y (t) = 1] = (1+ e7)/2, for t > 0. Next, let X, := Y(n) forn =0,1,.... 
Then {X,,n =0,1,...} is a Markov chain. Calculate 

(a) its one-step transition probability matrix, 


(b) the limiting probabilities of the chain, if they exist. 


Question no. 10 
The failures of a certain machine occur according to a Poisson process with 
rate A = | per week. 


(a) What is the probability that the machine will have at least one failure 
during each of the first two weeks considered? 


(b) Suppose that exactly five failures have occurred during the first four weeks 
considered. Let M be the number of failures during the fourth of these four 
weeks. Calculate E[M | M > 0]. 


Question no. 11 
Let {N(t),t > 0} be a Poisson process with rate A > 0. We define 


N,(t) = N(vt), No(t) = N(2t), and N3(t) = N(t +2) — N(2) 


and we consider the processes {Nx(t),t > 0}, for k = 1,2,3. Which of these 
stochastic processes is (or are) also a Poisson process? Justify. 


Question no. 12 

The power failures in a certain region occur according to a Poisson pro- 
cess with rate A; = 1/5 per week. Moreover, the duration X (in hours) of a 
given power failure has an exponential distribution with parameter Ag = 1/2. 
Finally, we assume that the durations of the various power failures are inde- 
pendent random variables. 


(a) What is the probability that the longest failure, among the first three 
power failures observed, lasts more than four hours? 


(b) Suppose that there has been exactly one power failure during the first 
week considered. What is the probability that the failure had still not been 
repaired at the end of the week in question? 


Question no. 13 

A machine is made up of two components that operate independently. 
The lifetime X; (in days) of component i has an exponential distribution with 
parameter 4;, for i = 1,2. 

Suppose that the two components are placed in series and that as soon 
as a component fails, it is replaced by a new one. Let N(t) be the number of 
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replacements in the interval (0,t]. We can show that the stochastic process 
{N(t),t > 0} is a Poisson process. Give its rate A. 


Indication. If Xo is the time between two successive replacements, then we 
can write that Xo = min{X1, X>}. 


Question no. 14 
We consider the process {X(t),0 <t < 1} defined by 


X(t) = N(t?)—t?N(1) forO<t<1 


where {N(t),¢ > 0} is a Poisson process with rate > 0. 

(a) Calculate the autocorrelation function, Rx (ty, t2), of the stochastic process 
{X(t),0<t <1} at t) =1/4 and tg = 1/2. 

(b) Calculate P[X(t) > 0| N(1) = 1], forO<t<1. 


Question no. 15 

Let {N(t),t > 0} be a Poisson process with rate X. Show that, given that 
N(t) =n, N(s) has a binomial distribution with parameters n and s/t, where 
0<s<t, forall A (> 0). 


Question no. 16 
Travelers arrive at a bus station from 6 a.m., according to a Poisson process 
with rate A = 1 per minute. The first bus leaves 7’ minutes after 6 a.m. 


(a) Calculate the mean and the variance of the number of travelers ready to 
board this bus if (i) J has an exponential distribution with mean equal to 15 
minutes and (ii) T’ is uniformly distributed between 0 and 20 minutes. 


(b) Calculate the average number of passengers on the bus if it leaves at 6:15 
and if its capacity is 20 passengers. 


Question no. 17 

We suppose that every visitor to a museum, independently from the oth- 
ers, moves around the museum for 7 minutes, where T’ is a uniform random 
variable between 30 and 90 minutes. Moreover, the visitors arrive according 
to a Poisson process with rate A = 2 per minute. If the museum opens its 
doors at 9 a.m. and closes at 6 p.m., find the mean and the variance of the 
number of visitors in the museum (i) at 10 a.m. and (ii) at time to, where to 
is comprised between 10:30 a.m. and 6 p.m. 


Question no. 18 

Suppose that events occur at random in the plane, in such a way that 
the number of events in a region R is a random variable having a Poisson 
distribution with parameter AA, where \ is a positive constant and A denotes 
the area of the region R. A point is taken at random in the plane. Let D be 
the distance between this point and the nearest event. Calculate the mean of 
the random variable D. 
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Question no. 19 

The various types of traffic accidents that occur in a certain tunnel, over 
a given time period, constitute independent stochastic processes. We suppose 
that accidents stopping the traffic in the northbound (respectively, south- 
bound) direction in the tunnel occur according to a Poisson process with rate 
a (resp., 8). Moreover, accidents causing the complete closure of the tunnel 
occur at rate y, also according to a Poisson process. Let Ty (resp., Ts) be 
the time during which the traffic is not stopped in the northbound (resp., 
southbound) direction, from the moment when the tunnel has just reopened 
after its complete closure. 
(a) Calculate the probability density function of the random variable Ty. 


(b) Show that 


PIT xy >t, Ts > 7]= e ot Br—-ymax{t,7} for t,7 > 0 


(c) Check your answer in (a) with the help of the formula in (b). 


Question no. 20 

Let {N*(t),t > 0} be the stochastic process that counts only the even 
events (that is, events nos. 2,4,...) of the Poisson process {N(t),t > 0} in 
the interval [0,t]. Show that {N*(t),t > 0} is not a Poisson process. 


Question no. 21 

Let {Ni(t),t > 0} and {No(t),t > 0} be two independent Poisson pro- 
cesses, with rates \; and Ag, respectively. We define N(t) = Ni(t) — No(t). 
(a) Explain why the stochastic process {N(t),t > 0} is not a Poisson process. 
(b) Give a formula for the probability P[N(t2) — N(t1) = nl], for tg >t; > 0 
and n € {0,+1,+2,...}. 
Question no. 22 

Suppose that {N(t),t > 0} is a Poisson process with rate \ > 0 and that 
S is a random variable having a uniform distribution on the interval {0, 2}. 
(a) Obtain the moment-generating function of the random variable N(t +S). 
Indication. If X has a Poisson distribution with parameter a, then Mx(t) = 
exp{a(e* — 1)}. 
(b) Calculate the mean and the variance of N(t +S). 
Question no. 23 
(a) Is the Poisson process {N(t),¢ > 0} an ergodic process? Justify. 
(b) Is the stochastic process {X(t),t > 0}, where X(t) := N(t)/t, mean 
ergodic? Justify. 


Question no. 24 
City buses arrive at a certain street corner, between 5 a.m. and 11 p.m., 
according to a Poisson process with rate \ = 4 per hour. Let T; be the 


5.7 Exercises 295 


waiting time, in minutes, until the first bus (after 5 a.m.), and let M4 be the 
total number of buses between 5 a.m. and 5:15 a.m. 

(a) Calculate the probability P[T, € I, M = 1], where I := (a, 6] is included 
in the interval {0, 15]. 

(b) What is the variance of the waiting time between two consecutive arrivals? 


(c) If a person arrives at this street corner every morning at 9:05 a.m., what 
is the variance of the time during which she must wait for the bus? Justify. 


Question no. 25 

A truck driver is waiting to join the traffic on the service road of a highway. 
The truck driver blocks the service road for four seconds when he joins the 
traffic. Suppose that vehicles arrive according to a Poisson process and that 
six seconds elapse, on average, between two consecutive vehicles. Answer the 
following questions by assuming that the truck driver makes sure that he has 
enough time to perform his maneuver before merging into the traffic. 


(a) What is the probability that the truck driver is able to join the traffic 
immediately on his arrival at the intersection with the service road? 


(b) What is the mathematical expectation of the gap (in seconds) between the 
truck and the nearest vehicle when the truck driver merges into the traffic? 


(c) Calculate the average number of vehicles that the truck driver must let go 
by before being able to merge into the traffic. 


Question no. 26 
Particles are emitted by a radioactive source according to a Poisson process 
with rate A = In5 per hour. 


(a) What is the probability that during at least one of five consecutive hours 
no particles are emitted? 


(b) Knowing that during a given hour two particles were emitted, what is the 
probability that one of them was emitted during the first half-hour and the 
other during the second half-hour of the hour in question? 


(c) In (b), if we know that the first particle was emitted over the first half- 
hour, what is the probability that the second particle was emitted during the 
first half-hour as well? 


Question no. 27 

A system is made up of two components. We suppose that the lifetime (in 
years) of each component has an exponential distribution with parameter A 
= 2 and that the components operate independently. When the system goes 
down, the two components are then immediately replaced by new ones. We 
consider three cases: 


1. the two components are placed in series (so that both components must 
function for the system to work); 


296 5 Poisson Processes 


2. the two components are placed in parallel (so that a single operating com- 
ponent is sufficient for the system to function) and the two components 
operate at the same time; 

3. the two components are placed in parallel, but only one component oper- 
ates at a time and the other is in standby. 


Let N(t), for t > 0, be the number of system failures in the interval {0, t]. 
Answer the following questions for each of the cases above. 


(a) Is {N(t),t > 0} a Poisson process? If it is, give its rate A. If it’s not, justify. 


(b) What is the average time elapsed between two consecutive system failures? 


Question no. 28 

Let {N(t),t > 0} be a Poisson process with rate \ = 2 per minute, where 
N(t) denotes the number of customers of a newspaper salesperson in the 
interval 0, t]. 
(a) Let Ty, for k = 1,2,..., be the arrival time of the kth customer. Calculate 
the probability density function of T;, given that Tz = s. 


(b) Calculate the probability that at least two minutes will elapse until the 
salesperson’s second customer arrives, from a given time instant, given that 
no customers have arrived in the last minute. 


(c) Suppose that the probability that a given customer is a man is equal to 
0.7, independently from one customer to another. Let M be the number of 
consecutive customers who are men before the first woman customer arrives, 
from some fixed time instant tg > 0. Calculate the mathematical expectation 
of M, given that the first customer after tg was a man. 


Question no. 29 

Suppose that {N(t),t > 0} is a Poisson process with rate \ = 2. 
(a) Let T, be the arrival time of the kth event of the process {N(t),t > O}, 
for k = 1,2,.... Calculate P[T, + Th < Ts]. 
(b) Let S be a random variable having a uniform distribution on the interval 
[0, 1] and that is independent of the process {N(t), t > 0}. Calculate E[N?(S)}. 
(c) We define X, = N(n?), for n = 0,1,2,.... Is the stochastic process 
{X,,n = 0,1,2,...} a Markov chain? Justify by calculating the probability 
P[Xn41 =j Xn = 1, Xn-1 = tn-1,--- , Xo a io]. 
Question no. 30 

We denote by N(t) the number of failures of a machine in the interval (0, t]. 
We suppose that N(0) = 0 and that the time 7 until the first failure has a 
uniform distribution on the interval (0, 1]. Similarly, the time 7-1; between 
the (k — 1)st and the &th failure has a U(0, 1] distribution, for k = 2,3,.... 
Finally, we assume that 7),7,,... are independent random variables. 


(a) Calculate (i) the failure rate of the machine and (ii) P[N(1) = 1]. 
(b) (i) Let 7 := -5 Int, for k = 0,1,... . We define (see Ex. 5.1.2) 
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n-1 
Spo =O and Suh forn > 1 
k=0 


Finally, we set 
N*(t) = max{n > 0: Sp < t} 


Show that the stochastic process {N*(t),t > 0} is a Poisson process, with 
rate A = 2, by directly calculating the probability density function of 7;, for 
all k, with the help of Proposition 1.2.2. 

(ii) Use the central limit theorem to calculate (approximately) the proba- 
bility P{S100 > 40}. 


Question no. 31 
In the preceding question, suppose that the random variables Tx all have 
the density function 


f(s) = se~* /? for s >0 


(a) (i) Now what is the failure rate of the machine and (ii) what is the value 
of the probability P[N(1) = 1]? 
Indication. We have 


a 
/ tet +dt ~ 0.5923 
0 


(b) (i) Let 7f := Te for k = 0,1,... . We define the random variables S, and 
the stochastic process {N*(t),t > 0} as in the preceding question. Show that 
the process {N*(t),¢ > 0} is then a Poisson process with rate \ = 1/2. 

(ii) Calculate approximately the probability P[S25 < 40] with the help of 
the central limit theorem. 


Question no. 32 

A woman working in telemarketing makes telephone calls to private homes 
according to a Poisson process with rate X = 100 per (working) day. We 
estimate that the probability that she succeeds in selling her product, on a 
given call, is equal to 5%, independently from one call to another. Let N(t) be 
the number of telephone calls made in the interval {0, t], where ¢ is in (working) 
days, and let X be the number of sales made during one day. 


(a) Suppose that the woman starts her working day at 9 a.m. and stops 
working at 7 p.m. Let 7 be the number of minutes between 9 a.m. and the 
moment of her first call of the day, and let Sg be the duration (in minutes) of 
this call. We suppose that Sg ~ Exp(1) and that 79 and So are independent 
random variables. What is the probability that the woman has made and 
finished her first call at no later than 9:06 a.m. on an arbitrary working day? 


(b) Calculate V[X | N(1) = 100]. 
(c) Calculate (approximately) P[N(1) = 100 | X = 5]. 
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re 2g: Ts 
Indication. Stirling’s* formula: n! ~ /2rn"t2e-". 
oOo 


(d) What is the probability that the woman will make no sales at all on exactly 
one day in the course of a week consisting of five working days? 


Question no. 33 

The breakdowns of a certain device occur according to a Poisson process 
with rate 4 = 2 per weekday, and according to a Poisson process (independent 
of the first process) with rate \ = 1.5 per weekend day. Suppose that exactly 
four breakdowns have occurred (in all) over two consecutive days. What is 
the probability that both days were weekdays? 


Question no. 34 

Let {N(t),¢ > 0} be a Poisson process with rate \ = 2 per minute. What 
is the probability that the time elapsed between at least two of the first three 
events of the process is smaller than or equal to one minute? 


Question no. 35 

Let N(t) be the number of telephone calls to an emergency number in the 
interval [0,t]. We suppose that {N(t),t > 0} is a Poisson process with rate » 
= 50 per hour. 
(a) What is, according to the model, the probability that there are more calls 
from 8 a.m. to 9 a.m. than from 9 a.m. to 10 a.m.? 
Indication. If X ~ Poi(50) and Y ~ Poi(50) are independent random variables, 
then we have that P[X = Y] ~ 0.0399. 
(b) Suppose that 20% of the calls are redirected to another service. Knowing 
that there were 60 (independent) calls during a given hour and that the first 
of these calls was redirected elsewhere, what is the mathematical expectation 
of the number of calls that were redirected over the hour in question? 


Question no. 36 
From a Poisson process with rate A, {N(t),t > 0}, we define the stochastic 
process {M(t),t > 0} as follows: 


M(t)=N(t)-t fort >0 


Calculate P[S, < 2], where S; := min{t > 0: M(t) > 1}. 


Question no. 37 
Let ae ae: 
X(t) := me Ni) fort >0 


where {N(t),¢ > 0} is a Poisson process with rate A and 6 > 0 is a constant. 
(a) Calculate E[X (t)]. 


4 See p. 93. 
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(b) Calculate Cov[X(s), X(t)], for s,t > 0. 
(c) Is the process {X(t),t > 0} wide-sense stationary? Justify. 


Question no. 38 

A machine is composed of three identical components placed in standby 
redundancy, so that the components operate (independently from each other) 
by turns. The lifetime (in weeks) of a component is an exponential random 
variable with parameter \ = 1/5. There are no spare components in stock. 
What is the probability that the machine will break down at some time during 
the next nine weeks, from the initial time, and remain down for at least a week, 
if we suppose that no spare components are expected to arrive in these next 
nine weeks? 


Question no. 39 
Let {X(t), t > 0} be the stochastic process defined by 


X(t) =tN(t) —[JN((t]) fort > 0 


where {N(t), t > 0} is a Poisson process with rate \ (> 0) and [¢] denotes the 
integer part of t. 

(a) Is the process {X(t),t > 0} a continuous-time Markov chain? Justify. 

(b) Calculate the probability p; ;(t,¢ + s) = P[X(t +s) = 7 | X(t) = 4], for 
1,9 € {0,1,...} and s,t > 0. 

Question no. 40 


We consider the stochastic process {X(t), ¢ > 0} defined from a Poisson 
process with rate A, {N(t),¢ > 0}, as follows (see Ex. 5.1.4): 


X(t)= N(t)—tN(1) for0<t<1 


(a) Let M be the number of visits of the stochastic process {X(t),t > 0} to 
any state x; > 0, from any state x; < 0, in the interval 0 < ¢ < 1. Calculate 
P|M =n | N(1) =n]. 

(b) We define T = min{t > 0: X(t) > 0}. Calculate the conditional probabil- 
ity density function fr(t | N(1) = 2). 


Question no. 41 

Consider the stochastic process {Y(t),¢ > 0} defined in Question no. 9, 
where {NV(t),¢ > 0} is now a Poisson process with an arbitrary rate A (> 0). 
It can be shown that we then have that P[Y(t) = 1] = (1+e7*)/2, for t > 0. 
Calculate 
a) P[N(2)—N(1) >1| N(1) = Jj, 
b) PIN(t) =0| Y(t) = 1], 
c) P[Y(s) =1| N(t) =1], where0 <5 <t. 
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Question no. 42 
We set 
A(t) = N(t+c)—N(e) fort >0 
where {N(t),t > 0} is a Poisson process with rate A and c is a positive 
constant. Is the stochastic process {X(t),t > 0} a Poisson process? Justify. 


Question no. 43 
Let {N(t),t > 0} be a Poisson process with rate A > 0 and let {X(t), 
t > 0} be the stochastic process defined as follows: 


for t > 0 


where 6 is a positive constant. 
(a) Calculate Cov[X (t,), X(t2)], for ty, tg > 0. 
(b) Is the process {X(t),t > 0} wide-sense stationary? Justify. 


Question no. 44 

Suppose that {N(t),t > 0} is a Poisson process with rate A. Let Xp := 
N?(n), forn = 0,1,... . Is the stochastic process {Xp,n = 0,1,... } a discrete- 
time Markov chain? If it is, give its one-step transition probability matrix. If 
it’s not, justify. 


Question no. 45 


Let 7}, be the arrival time of the kth event of the Poisson process {N(t), 
t > O}" with rate A, for k = 1,2,... . Calculate the covariance Cov([T}, T9]. 


Question no. 46 

We consider a Poisson process, {N(t),t > 0}, with rate A = 1. Let Th, 
Tg, ... be the arrival times of the events and let S; := T,, Sg := T2—-T, 
S3 := T3 — To, etc. 
(a) Calculate P[{S, < Sp < S3}U {$3 < S2 < S)}J. 
(b) Let X := S!/*. Calculate (i) fx(x) and (ii) PIX < $9]. 
(c) Calculate E[N?(1) | Ts = 5}. 


Question no. 47 

Let {X,,n = 0,1,...} be a (discrete-time) Markov chain whose state space 
is the set Z of all integers. Suppose that the process spends an exponential 
time 7 (in seconds) with parameter \ = 1 in each state before making a 
transition and that the next state visited is independent of r. Let N(t), for 
t > 0, be the number of transitions made in the interval [0, t]. 


(a) What is the probability that the third transition took place before the 
fifth second, given that five transitions occurred during the first 10 seconds? 


(b) Calculate E[|N(5) — 1]. 
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Sections 5.2 to 5.5 


Question no. 48 
The failures of a certain device occur according to a nonhomogeneous 
Poisson process whose intensity function A(t) is given by 


0.2if0<t<10 
MO= {O39 S10 


where ¢ is the age (in years) of the device. 


(a) Calculate the probability that a five-year-old device will have exactly two 
failures over the next 10 years. 


(b) Knowing that the device had exactly one failure in the course of the first 
5 years of the 10 years considered in (a), what is the probability that this 
failure took place during its sixth year of use? 


Question no. 49 

Suppose that traffic accidents occur in a certain region according to a 
Poisson process with rate A = 2 per day. Suppose also that the number M of 
persons involved in a given accident has a geometric distribution with param- 
eter p = 1/2. That is, 


PIM =m] =(1/2)™ for m=1,2,... 


(a) Calculate the mean and the variance of the number of persons involved in 
an accident over an arbitrary week. 


(b) Let T be the random variable denoting the time between the first person 
and the second person involved in an accident, from the initial time. Calculate 
the distribution function of T. 


Question no. 50 

Suppose that the monthly sales of a dealer of a certain luxury car constitute 
a conditional Poisson process such that A is a discrete random variable taking 
the values 2, 3, or 4, with probabilities 1/4, 1/2, and 1/4, respectively. 
(a) If the dealer has three cars of this type in stock, what is the probability 
that the three cars will be sold in less than a month? 


(b) Suppose that A = 3. Calculate V[M | M < 1], where M is the number of 
cars sold in one month. 


Question no. 51 

We suppose that the traffic at a point along a certain road can be described 
by a Poisson process with parameter \ = 2 per minute and that 60% of the 
vehicles are cars, 30% are trucks, and 10% are buses. We also suppose that the 
number K of persons in a single vehicle is a random variable whose function 
PK 1s 
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1/2ifk=1 
1/4 if k= 2 0.9ifk=1 
Px(k) = nae and One ae 
1/8 if k= 4 


in the case of cars and trucks, respectively, and px(k) = 1/50, for k = 
1,... ,50, in the case of buses. 


(a) Calculate the variance of the number of persons who pass by this point in 
the course of a five-minute period. 


Indication. We have 2 
ss 52 = n(n +1)(2n +1) 
~ 6 


(b) Given that five cars passed by the point in question over a five-minute 
period, what is the variance of the total number of vehicles that passed by 
that point during these five minutes? 


Indication. We assume that the number of cars is independent of the number 
of trucks and buses. 


(c) Calculate, assuming as in (b) the independence of the vehicles, the prob- 
ability that two cars will pass by this point before two vehicles that are not 
cars pass by there. 


(d) Suppose that actually 


t/5 if0<t<10 
A=X(t) = 2  if10<t<50 
(60 — t)/5 if 50<t < 60 


Calculate the probability P[N(60) = 100 | N(30) = 60], where N(¢) is the 
total number of vehicles in the interval [0, ¢]. 


Question no. 52 

Let N(t) be the number of accidents at a specific intersection in the interval 
[0,4]. We suppose that {N(t),¢ > 0} is a Poisson process with rate A, = 1 
per week. Moreover, the number Y; of persons injured in the kth accident has 
(approximately) a Poisson distribution with parameter Ag = 1/2, for all k. 
Finally, the random variables Y;, Yo,... are independent among themselves 
and are also independent of the stochastic process {N(t),¢ > O}. 


(a) Calculate the probability that the total number of persons injured in the 
interval [0, ¢] is greater than or equal to 2, given that N(t) = 2. 

(b) Calculate V[N(t)¥4]. 

(c) Let S, be the time instant when the kth person was injured, for k = 
1,2,... . We set T = Sp — Sj. Calculate P|T > 0]. 


Question no. 53 
Let {N(t),t > 0} be a Poisson process with rate L. 
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(a) Suppose that EL is a random variable having an exponential distribution 
with parameter 1 (so that {N(t),t > 0} is actually a conditional Poisson 
process). Let T, be the arrival time of the first event of the stochastic process 
{N(t),t > 0}. 

(i) Calculate the probability density function of T}. 

(ii) Does the random variable T, have the memoryless property? Justify. 


(b) Let L = 1/2. Calculate E[N?(1) | N(1) < 2]. 
(c) Let LD = 1. Calculate P{N(3) — N(1) > 0| N(2) = 1). 


Question no. 54 
Telephone calls to an emergency number arrive according to a nonhomo- 
geneous Poisson process whose intensity function is given by 


i= 2if0O<t<6 (by night) 
“| dif 6<t < 24 (by day) 


where ¢ is in hours, and A(t) = A(t — 24), for t > 24. Furthermore, the 
duration (in minutes) of a call received at night has a uniform distribution on 
the interval (0, 2], whereas the duration of a call received during the day has 
a uniform distribution on the interval (0, 3]. Finally, the durations of calls are 
independent random variables. 


(a) Calculate the probability that an arbitrary telephone call received at night 
will be longer than a given call received during the day. 

(b) Calculate the variance of the number of calls received in the course of a 
given week. 

(c) Let D be the total duration of the calls received during a given day. 
Calculate the variance of D. 


Question no. 55 

Let {N(t),t > 0} be a Poisson process with rate \ = 2. Suppose that all 
the events that occur in the intervals (2k, 2k + 1], where k € {0,1,2,...}, are 
counted, whereas the probability of counting an event occurring in an interval 
of the form (2k + 1,2k + 2] is equal to 1/2. Let N(t) be the number of events 
counted in the interval {0, t]. 


(a) Calculate P[N,(2.5) > 2]. 
(b) Calculate V{[N\ (2) — 2, (1)]. 


(c) Let S, be the arrival time of the first counted event. Calculate 


PIS: <s|Ni(2)=1] Vs€ (0,2 


Question no. 56 

In the preceding question, suppose that A = 1 and that the probability of 
counting an event occurring in an interval of the form (2k +1, 2k + 2] is equal 
to (1/2)*, for k € {0,1,2,...}. 
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a 


b 


(a) Calculate P[Ni(5) > 5 | Ni (2) = 2]. 

(b) Calculate Cov[Ny (5), N1(2)}. 

(c) Let M be the total number of counted events in the intervals of the form 
(2k +1, 2k + 2]. Calculate E{A]. 

(d) Let S be the arrival time of the first event in an interval of the form 
(2k, 2k +1]. Calculate fs(s | N(3) — N(2) + N(1) = 1). 


Question no. 57 
Let {Ni (t),t > 0} be a (homogeneous) Poisson process with rate A = 1, 
and let {No(t),t > 0} be a nonhomogeneous Poisson process whose intensity 


function is 
1ifO<t<1 
ney {oie 


We suppose that the two stochastic processes are independent. Moreover, let 
{Y (t),t > 0} be a compound Poisson process defined by 


Y(t)= 5) xX; — (and Y(0) =0 if Ni(t) = 0) 


where X; has a Poisson distribution with parameter 1, for all i. Calculate 
(a) P[Ti,2 < To2,Ni(1) = No(1) = 0], where Tm,» denotes the arrival time of 
the nth event of the process {Nm(t),t > 0}, form =1,2 andn =1,2,..., 
(b) V[N2(1)(N2(2) — No(1))), 

(c) (i) P[Y(1) < Ni(1) < JJ; (ii) V[Y(5) | Ni(5) < 2]. 

Question no. 58 


Suppose that the intensity function A(t) of the nonhomogeneous Poisson 
process {N(t),t > 0} is given by 


t 
where » > 0 and ¢ is in minutes. 


(a) Let T, be the arrival time of the first event of the stochastic process 
{N(t),t > 0}. Calculate P[T; < s | N(1) = 1], for s € (0, 1]. 

(b) Suppose that A = 2 and that N(5) > 2. Calculate the probability that, at 
time tp = 10, at least five minutes have elapsed since the penultimate event 
occurred. 


Question no. 59 
In the preceding question, suppose that 


for t > 0 


where t is in minutes. 
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(a) Calculate fr,(s | N(1) = 1), for s € (0, 1]. 

(b) Suppose that N(5) > 3. Calculate the probability that, at time to = 10, 
at least five minutes have elapsed since the antepenultimate (that is, the one 
before the penultimate) event occurred. 


Question no. 60 
The stochastic process {Y(t), t > 0} is a compound Poisson process defined 


by 
N(t) 


Y(t)= 5° X, (and Y(t) = 0 if N(¢) = 0) 
k=1 


where X, has a geometric distribution with parameter 1/4, for k = 1,2,..., 
and {N(t),t > 0} is a Poisson process with rate \ = 3. 


(a) Calculate E[Y(t) | Y(t) > 0], for t > 0. 


(b) Calculate approximately P[Y(10) > 100] with the help of the central limit 
theorem. 


Question no. 61 

Suppose that, in the preceding question, Xz is a discrete random variable 
such that P[X;, = 1] = P[X, = 2] = 1/2, fork =1,2,..., and that {N(t),t > 
0} is a Poisson process with rate \ = 2. 
(a) Calculate E[Y?(t) | Y(t) > 0], for t > 0. 


(b) Use the central limit theorem to calculate (approximately) the probability 
P[Y (20) < 50]. 


Question no. 62 

The (independent) visitors of a certain Web site may be divided into two 
groups: those who arrived on this site voluntarily (type I) and those who 
arrived there by chance or by error (type II). Let N(t) be the total number 
of visitors in the interval [0,t]. We suppose that {N(¢),t > O} is a Poisson 
process with rate \ = 10 per hour, and that 80% of the visitors are of type I 
(and 20% of type II). 


(a) Calculate the mean and the variance of the number of visitors of type I, 
from a given time instant, before a second type II visitor accesses this site. 


(b) Calculate the variance of the total time spent on this site by the visitors 
arrived in the interval (0, 1] if the time (in minutes) X; (respectively, X7,) that 
a type I (resp., type II) visitor spends on the site in question is an exponential 
random variable with parameter 1/5 (resp., 2). Moreover, we assume that X; 
and X77 are independent random variables. 


(c) Suppose that, actually, {N(t),¢ > 0} is a nonhomogeneous Poisson process 
whose intensity function is 


5 if0<t<7 
MO =| ooitre do 94 
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and A(t+24n) = X(t), for n = 1,2,... . Given that exactly one visitor accessed 
this site between 6 a.m. and 8 a.m., what is the distribution function of the 
random variable S denoting the arrival time of this visitor? 


Question no. 63 

At night, vehicles circulate on a certain highway with separate roadways 
according to a Poisson process with parameter \ = 2 per minute (in each di- 
rection). Due to an accident, traffic must be stopped in one direction. Suppose 
that 60% of the vehicles are cars, 30% are trucks, and 10% are semitrailers. 
Suppose also that the length of a car is equal to 5 m, that of a truck is equal 
to 10 m, and that of a semitrailer is equal to 20 m. 


(a) From what moment is there a 10% probability that the length of the queue 
of stopped vehicles is greater than or equal to one kilometer? 


(b) Give an exact formula for the distribution of the length of the queue of 
stopped vehicles after t minutes. 


Indication. Neglect the distance between the stopped vehicles. 


Question no. 64 

During the rainy season, we estimate that showers, which significantly 
increase the flow of a certain river, occur according to a Poisson process with 
rate \ = 4 per day. Every shower, independently from the others, increases 
the river flow during T days, where T is a random variable having a uniform 
distribution on the interval [3, 9]. 


(a) Calculate the mean and the variance of the number of showers that sig- 
nificantly increase the flow of the river 

(i) six days after the beginning of the rainy season, 

(ii) to days after the beginning of the rainy season, where tp > 9. 


(b) Suppose that every (significant) shower increases the river flow by a quan- 
tity X (in m/s) having an exponential distribution with parameter 1/10, 
independently from the other showers and from the number of significant 
showers. Suppose also that there is a risk of flooding when the increase in the 
river flow reaches the critical threshold of 310 m?/s. Calculate approximately 
the probability of flooding 10 days after the beginning of the rainy season. 


Question no. 65 

Independent visitors to a certain Web site (having infinite capacity) arrive 
according to a Poisson process with rate \ = 30 per minute. The time that a 
given visitor spends on the site in question is an exponential random variable 
with mean equal to five minutes. Let Y(t) be the number of visitors at time 
t > 0. The stochastic process {Y(t),¢ > 0} is a filtered Poisson process. 


(a) What is the appropriate response function? 
(b) Calculate E[Y(t)] and V[Y(t)]. 
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Question no. 66 
Let {Ni(t),t > 0} and {No(t),t > 0} be independent Poisson processes, 
with rates A, and A», respectively. 


(a) We denote by T2, the arrival time of the first event of the stochastic 
process {N2(t),t > 0}. Calculate E[Ny(T2,1) | Ni(T2,1) < 2]. 


(b) We define M(t) = N,(¢) + No(t) and we set 


1 if the kth event of the process {M(t),t > 0} 
is an event of {N,(t),t > 0} 
Xp = 
0 if the kth event of the process {M(t),t > 0} 
is an event of { No(t),t > 0} 


Calculate V[Y (t)], where 


M(t) 
Y(t):= $° X, (and Y(t) = 0 if M(t) = 0) 
k=1 


(c) Suppose that {N2(t),t > 0} is rather a nonhomogeneous Poisson process 
whose intensity function is 


1 if0<t<1 
datt)= {a aeet tesa 


Calculate P[T2,, < 2], where >, is defined in (a). 


Section 5.6 


Question no. 67 

A system is composed of two components that operate at alternate times. 
When component i, for i = 1,2, starts to operate, it is active during X; 
days, where X; is an exponential random variable with parameter A; and is 
independent of what happened previously. The state of the components is 
checked only at the beginning of each day. If we notice that component 1 
is down, then we set the other component going, and component i will be 
repaired (in less than one day). 
(a) Let N; be the number of consecutive days during which component i 
is responsible for the functioning of the system, for i = 1,2. What is the 
probability distribution of N;? 
(b) Suppose that the two components are identical. That is, A; = Az := A. At 
what rate do the components relieve each other (over a long period)? 
(c) If Ay = 1/10 and Az = 1/12, what proportion of time, when we consider a 
long period, is component 1 responsible for the functioning of the system? 
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Question no. 68 

A woman makes long-distance calls with the help of her cell phone accord- 
ing to a Poisson process with rate A. Suppose that for each long-distance call 
billed, the next call is free and that we fix the origin at the moment when a 
call has just been billed. Let N(t) be the number of calls billed in the interval 
(0, ¢]. 
(a) Find the probability density function of the random variables 79,71,..., 
where 79 is the time until the first billed call, and 7; is the time between the 
(i ~ 1)st and the ith billed call, for i > 1. 


(b) Calculate the probability P[N(t) =n], forn =0,1,.... 
(c) What is the average time elapsed at time ¢ since a call has been billed? 


Question no. 69 

A machine is made up of two independent components placed in series. 
The lifetime of each component is uniformly distributed over the interval [0, 1]. 
As soon as the machine breaks down, the component that caused the failure is 
replaced by a new one. Let N(t) be the number of replacements in the interval 
(0, t]. 
(a) Is the stochastic process {N(t),t > 0} a continuous-time Markov chain? 
Justify. 


(b) Is {N(t),¢ > 0} a renewal process? Justify. 


(c) Let S, be the time of the first replacement. Calculate the probability 
density function of $j. 


Question no. 70 (See Question no. 98, p. 171) 

The lifetime of a certain machine is a random variable having an expo- 
nential distribution with parameter 4. When the machine breaks down, there 
is a probability equal to p (respectively, 1 — p) that the failure is of type I 
(resp., II). In the case of a type I failure, the machine is out of use for an 
exponential time, with mean equal to 1/u time unit(s). To repair a type II 
failure, two independent operations must be performed. Each operation takes 
an exponential time with mean equal to 1/,. 

(a) Use the results on regenerative processes to calculate the probability that 
the machine will be in working state at a (large enough) given time instant. 


(b) What is the average age of the machine at time t? That is, what is the 
average time elapsed at time ¢ since the most recent failure has been repaired? 
Assume that A = pu. 


Question no. 71 
We consider a discrete-time Markov chain whose state space is the set 
{0, 1,2} and whose one-step transition probability matrix is 


0 1/21/2 
P=/1/2 0 1/2 
1 0 0 
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We say that a renewal took place when the initial state 0 is revisited. 


(a) What is the average number of transitions needed for a renewal to take 
place? 
(b) Let M(t), for t > 0, be the number of renewals in the interval (0,t], where 
t is in seconds. If we suppose that every transition of the Markov chain takes 
one second, calculate 

(i) the distribution of the random variable N(6.5), 

(ii) the probability P[.N(90) < 40] (approximately). 


Question no. 72 
We consider a birth and death process, {X(t),t > 0}, whose state space 
is the set {0,1} and for which 


Ao =A and fy = 


Moreover, we suppose that (0) = 0. We say that a renewal occurred when 
the initial state 0 is revisited. Let N(t), for t > 0, be the number of renewals 
in the interval {0, ¢]. 

(a) Let Tn_1 be the time between the (n — 1)st and the nth renewal, for 
n > 1 (to being the time until the first renewal). Find the probability density 
function of 7,_1 if A = p. 

(b) Calculate approximately the probability P[N(50) < 15] if A = 1/3 and 
=i. 

(c) Calculate the average time elapsed at a fixed time instant tp since a renewal 
occurred if A = 2 and fp = 1. 


Question no. 73 
Let {X(t),¢ > 0} be a birth and death process for which 


Aw=A Vn>0 and w=h, fp =O Vn>=2 


We suppose that X(0) = 0 and we say that a renewal took place when the 
process revisits the initial state 0. We denote by N(t), for t > 0, the number 
of renewals in the interval (0, ¢]. 
(a) Let T; be the time elapsed until the first renewal, and let M(t) be the 
number of deaths in the interval (0, t]. Calculate 

(i) E[Ti], 

(ii) Pflimyoo M(t) = ki], for k € {0,1,2,...}. 
(b) Is the stochastic process {X(t),t > 0} a regenerative process? Justify. 
(c) Suppose now that Ag = A and Ay, = 0 if n > 1. Calculate the proportion 
of time that the process {X(t),t > 0} spends in state 0, over a long period. 


Question no. 74 

Let {N(t),t > 0} be a renewal process for which the time 7 between the 
successive events is a continuous random variable whose probability density 
function is given by 
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fr(s)=se~* fors>0 
That is, 7 has a gamma distribution with parameters a = 2 and A= 1. 


(a) Show that the renewal function, my(t), is given by 


1 
mn(t) = ge +2t-1) fort >0 


Indication. We have 


9° yen-l 1 et he 
oS creme Set) 
n=1 . 


(b) Calculate approximately the probability P[Tiv(100)41 < 101.5}. 


(c) Let X(t) := (-1)%“), for t > 0. The stochastic process {X(t),t > 0} isa 
regenerative process. Calculate the limiting probability that X(t) = 1. 
Question no. 75 

Consider the renewal process {N(t),t > 0} for which the time 7 between 


the consecutive events is a continuous random variable having the following 
probability density function: 


fr(s)=2s forO<s<1 


(a) Calculate the renewal function, my(t), forO<t<1. 


Indication. The general solution of the second-order ordinary differential equa- 
tion y" (x) = ky(a) is 


1/2 _ 41/2 
y(x) = cye* tee 


where k £0, and cy and co are constants. 


(b) According to Markov’s inequality, what is the maximum value of the 
probability P[T(1/2)41 2 1]? 


(c) Suppose that we receive a reward of $1 per time unit when the age, A(t), of 
the renewal process is greater than or equal to 1/2, and of $0 when A(t) < 1/2. 
Calculate the average reward per time unit over a long period. 


Question no. 76 

The time between the successive renewals, for a certain renewal process 
{N(t),t > 0}, is a continuous random variable whose probability density 
function is the following: 


_ f1/2if0<s<1/2 
(ON agipere 
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(a) We can show that my(t) = e’/? — 1, for t € (0,1/2). Use this result to 
calculate my(t), for t € [1/2,1). 


Indication. The general solution of the ordinary differential equation 
y'(x) + P(x)y(2) = Q(x) 


is 


y(x) = et Pienta( Ff Q(ayel Pla)de gy + constant 


(b) Use the central limit theorem to calculate approximately P[T25 > 15], 
where Ts is the time of the 25th renewal of the process {N(t),t > 0}. 


(c) Let {X(t),t > 0} be a regenerative process whose state space is the set 
{1,2}. We suppose that the time that the process spends in state 1 is a random 
variable Y; such that 


_ f[l/2ifo<y<1/2 
PON opine 


while the probability density function of the time Y2 that the process spends 


in state 2 is / / 
_ [3/2if0<y<1/2 
fry) = ee if1/2<y<1 


Calculate limy., P[X(t) = 1]. 


Question no. 77 

We consider a system composed of two subsystems placed in series. The 
first subsystem comprises a single component (component no. 1), whereas the 
second subsystem comprises two components placed in parallel. Let 5; be the 
lifetime of component no. k, for k = 1,2,3. We assume that the continuous 
random variables S; are independent. 


(a) Let N(t) be the number of system failures in the interval [0, t]. In what 
case(s) will the stochastic process {N(t),t > 0} be a renewal process if the 
random variables 5; do not all have an exponential distribution? Justify. 


(b) In the particular case when the variable S;, has an exponential distribution 
with parameter 4 = 1, for all k, the process {N(t),t > 0} ts a renewal 
process. Calculate the probability density function of the time 7 between two 
consecutive renewals. 


Question no. 78 
Suppose that {N(t),t > 0} is a renewal process for which the time 7 
between two consecutive renewals is a continuous random variable such that 


fey= | aoa Oe #1 


0 elsewhere 


312 5 Poisson Processes 


(a) Calculate the renewal function, my(t), for0<t <1. 
Remark. See the indication for Question no. 76. 


(b) If we receive a reward of $1 at the moment of the nth renewal if the 
duration of the cycle has been greater than 1/2 (and $0 otherwise), what is 
the average reward per time unit over a long period? 


Question no. 79 

We consider a system made up of two subsystems placed in parallel. The 
first subsystem is composed of two components (components nos. 1 and 2) 
placed in parallel, while the second subsystem comprises a single component 
(component no. 3). Let S, denote the lifetime of component no. k, for k = 
1, 2,3. The continuous random variables S; are assumed to be independent. 


(a) Suppose that components nos. 1 and 2 operate at the same time, from the 
initial time, whereas component no. 3 is in standby and starts operating when 
the first subsystem fails. When the system breaks down, the three components 
are replaced by new ones. Let N(t), for t > 0, be the number of system failures 
in the interval {0,t]. Then {N(t),t > 0} ts a renewal process. Let 7 be the 
time between two consecutive renewals. Calculate the mean and the variance 
of 7 if S, ~ U(0,1), for k = 1,2,3. 

(b) Suppose that we consider only the first subsystem and that the two com- 
ponents are actually placed in series. When this subsystem fails, the two 
components are replaced by new ones. As in (a), the process {N(t),t > 0} 
is a renewal process. Calculate the renewal function mn(t), for 0 < ¢ < 1, if 
Sp ~ U(0,1), for k = 1,2. 


Indication. The general solution of the differential equation 
y" (x) — 2y'(x) + 2y(a) +2 =0 
is 
y(z) = —1+ ce” cosa + coe” sing 
where c; and c2 are constants. 


(c) Suppose that in (b) we replace only the failed component when the sub- 
system breaks down and that S; ~ Exp(2), for k = 1, 2. 

(i) Calculate the mean of Ty (t)41 — t. 

(ii) Deduce from it the value of my(é), for t > 0. 


Question no. 80 

Is a nonhomogeneous Poisson process with intensity function A(t) = ¢, 
for all t > 0, a renewal process? If it is, give the distribution of the random 
variables 7;,. If it’s not, justify. 


Question no. 81 

Use the renewal equation to find the distribution of the random variables 
Tr, taking their values in the interval (0,7/2], of a renewal process for which 
my(t) = t?/2, forO<t < 7/2. 
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Question no. 82 

Let {X(t),t > 0} be a birth and death process with state space {0, 1,2} 
and having the following birth and death rates: Ay = A1 = A > 0, #4, unknown, 
and fig = ps > 0. For what value(s) of j4; is the process regenerative? Justify. 


Question no. 83 ; 

Suppose that {N(t),t > 0} is a Poisson process with rate A > 0. For a 
fixed time instant t > 0, we consider the random variables A(t) and D(t) (see 
p. 280). 

(a) Calculate the distribution of both A(t) and D(t). 

(b) What is the distribution of A(t) + D(t)? 

(c) Suppose now that we consider the Poisson process over the entire real line, 
that is, {N(t),¢ € R}. What then is the distribution of A(t) + D(t)? 

(d) In (c), we can interpret the sum A(t) + D(t) as being the length of the 
interval, between two events, which contains the fixed time instant t. Explain 
why the distribution of this sum is not an exponential distribution with pa- 
rameter A. 


Question no. 84 

Let {B(t),¢ > 0} bea standard Brownian motion, that is, a Wiener process 
with drift coefficient 4 = 0 and diffusion coefficient o? = 1. Suppose that 
when B(t) = a (> 0) or b = —a, the process spends an exponential time with 
parameter A = 2 in this state (a or b = —a). Then, it starts again from state 
0. 


(a) What fraction of time does the process spend in state a or b = —a, over a 
long period? 


Indication. Let mi(x) be the average time that the process, starting from 
x € (b,a), takes to reach a or b. We can show that the function m1(z) satisfies 
the ordinary differential equation 


sia) Snel (with m,(b) = m,(a) = 0) 


(b) Answer the question in (a) if b = —oo rather than —a. 


Question no. 85 

Suppose that the time between two consecutive events for the renewal process 
{N(t), t > 0} is equal to 1 with probability 1/2 and equal to 2 with probability 
1/2, 

(a) Give a general formula for the probability P[N(2n) = k], where n and k 
are positive integers. 

(b) Calculate my(3). 


(c) Let Z(t) := 1 if N(t) = [t] and I(t) = 0 otherwise, where [| |] denotes the 
integer part. Calculate the variance of I(t). 


6 


Queueing Theory 


6.1 Introduction 


In this chapter, we will consider continuous-time and discrete-state stochastic 
processes, {X(t),t > 0}, where X(t) represents the number of persons in 
a queueing system at time t. We suppose that the customers who arrive in 
the system come to receive some service or to perform a certain task (for 
example, to withdraw money from an automated teller machine). There can 
be one or many servers or service stations. The process {X(t),t > 0} is a 
model for a queue or a queueing phenomenon. If we want to be precise, the 
queue should designate the customers who are waiting to be served, that is, 
who are queueing, while the queueing system includes all the customers in 
the system. Since queue is the standard expression for this type of process, 
we will use these two expressions interchangeably. Moreover, it is clear that 
the queueing models do not apply only to the case when we are interested in 
the number of persons who are waiting in line. The customers in the system 
may be, for example, airplanes that are landing or are waiting for the landing 
authorization, or machines that have been sent to a repair shop, etc. 

Kendall’ proposed, in a research paper published in 1953, a notation to 
classify the various queueing models. The most general notation is of the form 
A/S/s/c/p/D, where 


A denotes the distribution of the time between two successive arrivals, 
S denotes the distribution of the service time of customers, 

s is the number of servers in the system, 

c is the capacity of the system, 

p is the size of the population from which the customers come, 

D designates the service policy, called the discipline, of the queue. 


' David George Kendall, retired professor of the University of Cambridge, in Eng- 
land. 
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We suppose that the times 7, between the arrivals of successive customers 
are independent and identically distributed random variables. Similarly, the 
service times S,, of the customers are random variables assumed to be i.i.d. 
and independent of the 7,,’s. Actually, we could consider the case when these 
variables, particularly the S,,’s, are not independent among themselves. 

The most commonly used distributions for the random variables 7, and 
Sn, and the corresponding notations for A or S, are the following: 


M exponential with parameter X or ju; 
E,, Erlang (or gamma) with parameters k and X or pi; 
D degenerate or deterministic (if T, or Sy is a constant); 


G general (this case includes, in particular, the uniform distribution). 


Remarks. i) We write M when 7, (respectively, S,,) has an exponential distri- 
bution, because the arrivals of customers in the system (resp., the departures 
from the system in equilibrium) then constitute a Poisson process, which is a 
Markovian process. 


ii) We can use the notation GI for general independent, rather than G, to be 
more precise. 


The number s of servers is a positive integer, or sometimes infinity. (For 
example, if the customers are persons arriving in a park and staying there 
some time before leaving for home or elsewhere, in which case, the customers 
do not have to wait to be served.) 

By default, the capacity of the system is infinite. Similarly, the size of the 
population from which the customers come is assumed to be infinite. If ¢ (or 
p) is not equal to infinity, its value must be specified. On the other hand, 
when c = p = 00, we may omit these quantities in the notation. 

Finally, the queue discipline is, by default, that of first-come, first-served, 
which we denote by FCFS or by FIFO, for first-in, first-out. We may also 
omit this default discipline in the notation. In all other cases, the service 
policy used must be indicated. We can have LIFO, that is, last-in, first-out. 
The customers may also be served at random (RANDOM). Sometimes one or 
more special customers are receiving priority service, etc. 

In this book, except in the penultimate subsection of the current chap- 
ter and some exercises, we will limit ourselves to the case when the random 
variables T, and S,, have exponential distributions, with parameters A and p, 
respectively. That is, we will only study models of the form M/M. Moreover, 
in the text, the service policy will be the default one (FIFO). However, in the 
exercises, we will often modify this service policy. 

For all the queueing systems that we will consider, we may assume that 
the limit 


Tr = lim P[LX(t) =n] (6.1) 


t—00 
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exists, for all n > 0. Thus, 7, designates the limiting probability that there 
are exactly n customers in the system. Moreover, 7, is also the proportion of 
time when the number of customers in the system is equal to n, over a long 
period. 

The quantities of interest when we study a particular queueing system 
are above all the average number of customers in the system, when it is in 
equilibrium (or in stationary regime), and the average time that an arbitrary 
customer spends in the system. We introduce the following notations: 


N is the average number of customers in the system (in equilibrium); 
No is the average number of customers who are waiting in line; 

Ng is the average number of customers being served; 

T is the average time that an arbitrary customer spends in the system; 
Q is the average waiting time of an arbitrary customer; 


S is the average service time of an arbitrary customer. 


Often, we must content ourselves with expressing these quantities in terms 
of the limiting probabilities 7,. Moreover, notice that N = Ng + Ng and 
TH=Q+5S. 

Let N(t), for t > 0, be the number of customers who arrive in the system 
in the interval (0, t]. Given that the random variables 7, are independent and 
identically distributed, the process {N(t),t > 0} is a renewal process. We 
denote by Aq the average arrival rate of customers in the system. That is (see 
Prop. 5.6.7), 

N(t) 1 


= lm — = —— 6.2 
Xe = lim ; Ein (6.2) 


Remarks. i) If Tt, ~ Exp(A), then we obtain A, = 4. 


ii) We can also define A., namely, the average entering rate of customers into 
the system. If all the arriving customers enter the system, then Ae = a. 
However, if the capacity of the system is finite, or if some customers refuse 
to enter the system if it is too full when they arrive, etc., then A. will be 
smaller than Ag. Suppose, for example, that the customers arrive according 
to a Poisson process with rate A. If the system capacity is equal to c (<oo) 
customers, then we may write that the rate A. is given by » (1 —-,). Indeed, 
in this case, (1 — 7,) is the (limiting) probability that an arriving customer 
will enter the system. 


We can establish a relation between the quantities N and T by using a 
cost equation. Suppose that the customers entering the system pay a certain 
amount of money. Let A, be the average earning rate of the system. Then, 
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after a long enough time to, the average amount of money the system earns 
is approximately equal to A, - to. On the other hand, this quantity is also 
approximately equal to I-A, -to, where M is the average amount of money 
a customer who enters the system pays, and A, is defined above. The equality 
between the two expressions for the average amount of money the system 
earns, over a long period, can be justified rigorously. We then obtain the 
following proposition. 


Proposition 6.1.1. We have 
Ag = de is M (6.3) 


Corollary 6.1.1. If the customers pay $1 per time unit that they spend in the 
system (waiting to be served or being served), then Eq. (6.3) becomes 


N=\.-T (6.4) 


Remarks. i) The formula above is known as Little’s? formula. 


ii) Little’s formula may be rewritten as follows: if ¢ is large enough (for the 
process to be in equilibrium), then we have 


E[X(t)] = EIT] (6.5) 


where T is the total time that an arbitrary customer who enters the system will 
spend in this system. Actually, Eq. (6.5) is valid under very general conditions, 
in particular, for all the systems studied in this book, but in some cases it is 
not correct. Moreover, we can prove the following result: 


im 2 f° xq) dt = FIX) = XE (6.6) 
0 


tooo tg 


That is, the stochastic process {X(t),t > 0} is, in the cases of interest to us, 
mean ergodic (see p. 56). 


iii) If every customer pays $1 per time unit while being served (but not while 
she is waiting in queue) instead, then we obtain 


Ng = Xe : 5 (6.7) 
It follows that we also have 


Ng =r: @ (6.8) 


When the times between the arrivals of successive customers and the ser- 
vice times of customers are independent exponential random variables, the 


? John D.C. Little, professor at the Sloan School of Management of the Mas- 
sachusetts Institute of Technology, in the United States. 
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process {X(t),t > 0} is a continuous-time Markov chain. Moreover, if we as- 
sume (which will be the case, in general) that the customers arrive one at a 
time and are served one at a time, {X(t),t > 0} is then a birth and death 
process. We may therefore appeal to the results that were proved in Chapter 
3 concerning this type of process, particularly Theorem 3.3.4, which gives us 
the limiting probabilities of the process. 


6.2 Queues with a single server 


6.2.1 The model M/M/1 


We first consider a queueing system with a single server, in which the cus- 
tomers arrive according to a Poisson process with rate A, and the service times 
are independent exponential random variables, with mean equal to 1/1. We 
suppose that the system capacity is infinite, as well as the population from 
which the customers come. Finally, the queue discipline is that of first-come, 
first-served. We can therefore denote this model simply by M/M/1. 


The stochastic process {X(t),t > 0} is an irreducible birth and death 
process. We will calculate the limiting probabilities 7,, as we did in Chapter 3. 
The balance equations of the system (see p. 140) are the following: 


state 7 departure rate from j = arrival rate to j 


0 Ato = LT 
n (2 1) (A+ p)tn = At—1 + UTn41 


We have (see p. 141) 


IIg:=1 and , = 4 = (3) for n = 1,2,... (6.9) 
Mss fb ML 
— ae 
TUX 


If A < ps, the process {X(t),¢ > 0} is positive recurrent, and Theorem 3.3.4 
then enables us to write that 


a ae ig 2 orn= 
. prey (1 — (A/p)|7! f 0,1,... (6.10) 


Tn = (3) (1-2) Vn>0 (6.11) 
7 jh 


_ We can now calculate the quantities of interest. We already know that 
S = 1/p. Moreover, because here Ae = A, we may write that 


That is, 
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Fig. 6.1. State-transition diagram for the model M/M/1. 


Ns =2A.°S = 7 (6.12) 


Remarks. i) Since there is only one server, we could have directly calculated 
Ng as follows: 


Ns=1-m=1- (1-4) =2 (6.13) 
bb 

because the random variable Ng denoting the number of persons who are being 
served, when the system is in equilibrium, here has a Bernoulli distribution 
with parameter po := 1— 70. 

ii) The quantity p := A/p is sometimes called the traffic intensity of the sys- 
tem, or the utilization rate of the system. We see that the limiting probabilities 
exist if and only if p < 1, which is logical, because this parameter gives us the 
average number of arrivals in the system during a time period corresponding 
to the mean service time of an arbitrary customer. If p > 1, the length of the 
queue increases indefinitely. 


iii) From the balance equations of the system, we can draw the corresponding 
state transition diagram (and vice versa). Each possible state is represented by 
a circle, and the possible transitions between the states by arrows. Moreover, 
we indicate above or under each arrow the rate at which the corresponding 
transition takes place (see Fig. 6.1). 


Next, we have 


Een EC OY 


(3) aes ee (6.14) 


(6.15) 
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We deduce from this formula that 


(6.16) 
which implies that 


— = (6.17) 


Generally, it is difficult to find the exact distribution of the random variable 
T that designates the total time that an arriving customer will spend in the 
system (so that T = E[T)). For the M/M/1 queue, we can explicitly determine 
this distribution. To do so, we need a result, which we will now prove. 


Proposition 6.2.1. Let a, be the probability that an arbitrary customer finds 
n customers in the system (in equilibrium) upon arrival. If the customers 
arrive according to a Poisson process, then we have 


In =n Vn>0 (6.18) 


Proof. It suffices to use the fact that the Poisson process has independent 
ancrements. Suppose that the customer in question arrives at time t. Let F. = 
the customer arrives in the interval [t,t + €). We have 


P[{X(t7) =n} NF. 


Gece gee im RNa) ete Bal Rasen a PIF; 


PIF. | X(t~) = n|P[X(t-) =n] 


= yim elo PIF. 

ind... PIF JP(X(t7) =n]. ee 

See eR, ee Pe 

= Jim P[X(t")=nl=m O (6.19) 
—00 


Remark. It can also be shown that, for any system in which the customers 
arrive one at a time and are served one at a time, we have 


Qpedae Vn 20 (6.20) 


where 6, is the long-term proportion of customers who leave behind them n 
customers in the system when they depart. This follows from the fact that 
the transition rate from n to n + 1 is equal to the transition rate from n + 1 
to n, over a long period. 
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Proposition 6.2.2. For an M/M/1 queue, we have T ~ Exp(p — 4). 


Proof. Let R be the number of customers in the system when a new customer 
arrives. We have 


a Se SPIT <t|R=r|P[R=r] (6.21) 
r=0 


Moreover, by the memoryless property of the exponential distribution, we may 
write that 


T | {R=r}~G(r+1,p) (6.22) 


Finally, since the customers arrive according to a Poisson process, we have 
(by the preceding proposition) 


Preise (2) (1 = *) (6.23) 


It follows that 


PIT <t/= > [ porno lta as (3) (: - D 


(6.24) 


\| 
Me 
om 
See 
b~) 
» 
See 
| 
r 
tv) 
ZS 
a 
Qu 


Interchanging the summation and the integral, we obtain 


PIT <4= fw — A)e™#S s sas ds = fu — je“ 4-5 ds (6.25) 
r=0 . 


—S 


ers 


= fr(t)=(u—A)e™“—™* fort >0 O (6.26) 


We can also calculate the distribution of the waiting time, Q, of an arbi- 
trary customer arriving in the system. This random variable is of mixed type. 
Indeed, if the customer arrives while the system is empty, we have that Q = 0. 
On the other hand, if there are R = r > 1 customer(s) in the system upon his 
arrival, then Q has a G(r, 1) distribution. Since P[R = 0] = m9 = 1—A/p, by 
proceeding as above, we find that 


Me 


PIQ< t= (Q<t|R=r]P[R=r| (6.27) 


1-()+ (3 \( 1—e7 ) =1- (2) lets 


r=0 
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Thus, we have P[(Q = 0] = 1 —A/p and 
PO <Q<t]= (2) (1 = gotet) ift >0 (6.28) 
Bh 


Moreover, we calculate 


PO<Q<t] 
PIQ>d) ~ 
That is, Q | {Q > 0} ~ Exp(y — A). We may therefore write that 


PIQ<t|Q>0= 1—e HA) ift>0 (6.29) 


QI{Q>0}57 (6.30) 
which follows directly from the fact that 
x\' d 
a nee r r—-1 BN 
pin-rjr>q=—_ - t) 1-4) arc 


To X/b Hl bt 
=T,-1=P[R=r-1| Vr>1 (6.31) 


Remark. Since the random variables Q and S are independent (by assump- 
tion), we can check that T has an exponential distribution with parameter 
ft — A by convoluting the probability density functions of Q and S. Because 
Q is a mixed-type random variable, it is actually easier to make use of the 
formula (with p = A/1) 


foe) t 
Fr(t) = f Foa(t — s) fs(s) ds =| € —p ee VN) pe #* ds 


t 
=1]-—e7H_ pa ey e** ds 
0 


=1—e He HME eM) =1-e HM? EDO (6.32) 


The variance of the random variable N designating the number of cus- 
tomers present in the system in equilibrium (and whose mean value is N) is 
easily obtained by noticing [see Eq. (6.11)] that 


N+1~ Geom(1 — p) (6.33) 


It follows that 
p Au 
Vini= Vii (6.34) 
ee a ar eae 
Note that the mean N can be expressed as follows: 


ss 1 p 
= = —— —-] = —— 6.35 
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In the case of the random variable Ns, we mentioned above that it has a 
Bernoulli distribution with parameter 1 — 7 = p. Then we directly have 


VIN] = o(1 - p) = = ( : *) (6.36) 


To obtain the variance of Ng, we will use the following relation between 
Ng and N: 


0 ifN=0Oorl 
Noms yao een) 
It follows that 
oo oo 
E(N@] = Bis m1 = p? > ke? mg-1 = PP E|Z"] (6.38) 
k=1 k=1 


where Z ~ Geom(1 — p), from which we deduce that 


anal (cat azz) ee) 


Since the mean value of Ng [see Eq. (6.15)] is given by 


2 p" 


E(No| = ————~ = —— 6.40 
we obtain that 
1+p po p?(1 + p— p”) 
VIN =F (pth,)- o-oo) 6.41 
al= 0" \ a=?) 0-0? Oo ee 
Remark. Given that N = Ng + Ns, we can use the formula 
V[N] = V[Noe] + V[Ns| +2 Cov[Ng, Ns} (6.42) 


to calculate the covariance (and then the correlation coefficient) of the random 
variables Ng and Ng. 


Now, we have shown that T ~ Exp(j: — A). We then have 


(6.43) 


Similarly, S ~ Exp() (by assumption), so that V[S] = 1/7. Finally, by 
independence of the random variables Q and S, we obtain that 


_ 2 
= pA ae = 2HA = rT (6.44) 
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Remark. We can calculate V{[Q] without making use of the independence of 
Q and S. We deduce indeed from the formula (6.30) that 


EIQ? = B1Q? | Q > O}PIQ > 0] = EIT*\(1 —m) = EIT*p = 
(6.45) 
from which we obtain that 
vig “2? 22 ry) 


: (ud)? p2(u — A)? p?(u — A)? 


Example 6.2.1. Let K be the number of arrivals in the system during the 
service period of an arbitrary customer. We will calculate the distribution of 
K. First, we have 

uta 

where the service time S ~ Exp(jz) and the time 7 needed for a customer to 
arrive, having an exponential Exp(A) distribution, are independent random 
variables. Then, by the memoryless property of the exponential distribution, 
we may write that 


k 
le be 
Pi\K =k)=(1- for k = 1,2,... 
| | ( =) (x) " ‘ 


m 
K+1-~ Geom (45) 


P[K = 0] = P[S <7] = 


That is, 


from which we deduce that 


E|K| = (2**)-1-4 
Mb ft 

Thus, the average number of arrivals during the service period of a given 

customer is equal to the average number of arrivals in the course of a period 

corresponding to the mean service time of an arbitrary customer (see p. 320). 

This result is also easily proved as follows: 


E[K] = E[E[K | $I] = E[\S] = E[S] = i 


Example 6.2.2. The conditional distribution of the random variable N, namely, 
the number of customers in the system in stationary regime, knowing that 
N <™, is given by 


pu(n| N<m)= 0 diet VE for n =0,1,...,m 


7 eco T = Dopeo(A/H)* 
Note that the condition N < m does not mean that the system capacity is 
equal to m. It rather means that, at a given time instant, there were at most 
m customers in the system (in equilibrium). 
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Example 6.2.3. The queueing system M/M/1 is modified as follows: after hav- 
ing been served, an arbitrary customer returns to the end of the queue with 
probability p € (0,1). 
Remark. A given customer may return any number of times to the end of the 
queue. 
(a) (i) Write the balance equations of the system. 

(ii) Calculate, if they exist, the limiting probabilities 7;. 
(b) Calculate, in terms of the 7;’s (when they exist), 

(i) the average time that an arbitrary customer spends in the system, 


(ii) the variance of the number of customers in the system, given that it 
is not empty. 


Solution. (a) (i) The balance equations of the system are the following: 


state 7 departure rate from j = arrival rate to 7 


0 Ato = “(1 — p)m 
n (2 1) [A+ (1 — p)|t = Atn-1 + w(L — P)Tnr41 


(ii) The process considered is an irreducible birth and death process. We 
first calculate 


fora) 0° k 
AoA Ag=1 A 
S — Ol —_—_—_—_—— 
Sal grrr 2 TF 
The infinite sum converges if and only if A < (1 — p). In this case, we find 


that 
r 


~ w(l—p)-A 
Then, the 7;’s exist and are given by 


n= ama (Mga) = (Gets) Gate) 


fori =0,1,.... 

Remark. The result is obtained at once by noticing that the process considered 
is equivalent to an M/M/1 queueing system for which the service rate is 
(1—p)p. 

(b) (i) We seek T’. By the preceding remark, we may write (see the formula 
(6.17), p. 321) 


Sj 


ait 2s 
apy 


(ii) Let N be the number of customers in stationary regime. We have (see 
the formula (6.33), p. 323) 


T= 
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N+1~ Geom (1 =" — ) 

“(1 — p) 
Let M := N | {N > O}. We seek V[M]. By the memoryless property of the 
geometric distribution, we can write that Mf ~ Geom (2 ~ a =): It follows 


that 
a, A YP dud =p) 
VIM =o / (2 a) = ian 


6.2.2 The model M/M/1/c 


Although the 14/M/1 queue is very useful to model various phenomena, it 
is more realistic to suppose that the system capacity is an integer c < oo. 
For j = 0,1,... ,¢—1, the balance equations of the system remain the same 
as when c = oo. However, when the system is in state c, it can only leave it 
because of the departure of the customer being served. In addition, this state 
can only be entered from c — 1, with the arrival of a new customer. We thus 


have 
state 7 departure rate from 7 = arrival rate to j 


0 Ato = UT 
l<k<e-1 (A + )ap = ATR + UTR +1 
c BT = ATe~1 


The process {X(t),t > 0} remains a birth and death process. Moreover, given 
that the number of states is finite, the limiting probabilities exist regardless 
of the values the (positive) parameters A and ju take. 

As in the case when the system capacity is infinite, we find (see p. 319) 
that 


d k 
I = (3) fork =0,1,...,¢€ (6.47) 


It follows, if p:= A/p #1, that 
c c d k c ‘ lic pert 
k=0 k=0 k=0 


When A = yu, we have that p= 1, Ij, =1, and 7,5 Hk =¢+1, from which 
we calculate 


Pil p 
y Tope vee 
"Wa 5 1 = (6.49) 
k=0 **k 1 . 
aa P= 


for 7 =0,1,...,¢. 
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Remarks. i) We see that if X = y, then the c+ 1 possible states of the sys- 
tem in equilibrium are equally likely. Moreover, when c tends to infinity, the 
probability 7; decreases to 0, for every finite 7. This confirms the fact that, 
in the M/M]/1/00 model, the queue length increases indefinitely if \ = jy, so 
that there is no stationary regime. 

ii) If A > p, the limiting probabilities exist. However, the larger the ratio p = 
A/ is, the more 7, increases to 1 (and 7; decreases to 0, for j = 0,1,... ,e—1), 
which is logical. 

iii) Even if, in practice, the capacity c cannot be infinite, the 14/M/1 model 
is a good approximation of reality if the probability 7, that the system is full 
is very small. 


With the help of the formula (6.49), we can calculate the value of N. 


Proposition 6.2.3. In the case of the M/M/1/c queue, the average number 
of customers in the system in equilibrium is given by 


p e+1)pct} 
pal dee - Cee itp A 


Nat tae: “1 (6.50) 
c/2 ifp=1 
Proof. First, when p = 1, we have 
c c 1 
~ 1 1 1 c(ce+1)_e¢ 
N = —_—— = = => 6.51 
Daas cri 2." c+1 2 2 ( ) 


When p # 1, we must evaluate the finite sum 


k c 
r pr(1 — p) 1 k 
N= Dob ype = Top 2 a (5:62) 


Let X := Z —1, where Z has a geometric distribution with parameter 1 — p. 
We find that the probability mass function of X is given by 


px(k) = p*(1—p) fork =0,1,... (6.53) 
It follows that 
oo foe) 1 0 
key yp) — K(1 — 9) = —— -1= — 6.54 
dP (1—p)=1 and Date Aa i, (6.64) 


Making use of these formulas, we may write that 


Soh (Lope es S kp*(1—p (6.55) 


k=c+1 
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and 
Lo.) CO 
> kekK(1—p) = So F-(e +1) + (e+ lo*( - 9) 
k=c+1 k=c+1 
CO [o) 
= S> [k- (c+ 1)]p*(1 — p) ean, (c+ 1)p*(1 — p) 
=c+1 k=c 
= pt S mp™(1 — p) + (e+1)p*1 s p™(1—p) 
m=0 
~ pee + (c+ 1p (6.56) 
so that 
= 1 p p 
N= _ acti = 1 ct1i 
rami p l=9 (c+ )p 


Remarks. i) We easily find that 


lim N = 1 
c™oo 


which corresponds to the results obtained in the preceding subsection. 


ors 

fp<l 
pen (6.58) 
co ifp>1 


ii) When the capacity ¢ of the system is very limited (for example, when 
c = 2,3, or 4), once we have calculated the 7;’s, we can directly obtain N 
from the definition of the mathematical expectation of a discrete random 
variable: N := )7y_k ™. We can then also calculate the variance of the 
random variable N as follows: 


V[N] := sk — E(N])?x, = 5° k? a, — (E[N])? (6.59) 


k=0 k=0 


iii) When p = 1, we find, using the formula 


that 


3 . 4 c(2c +1) 
= 3 = 6.61 
E(N?] 2 a = Sse ; (6.61) 
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so that 


c(2e+1) e\2  c(e+2) 
UMS eg Gg) =a coe) 


Now, as in the case of the M/M/1/o0 queue, we have 


Cc 
Ng = 01-1 =1- 7 (6.63) 
k=1 
and 


Next, the average entering rate of customers into the system (in equilib- 
rium) is given by 


pene (cee (6.65) 


because the customers always arrive according to a Poisson process with rate 
A but can enter the system only if it is not full (or saturated), that is, if it is 
in one the following states: 0,1,...,¢c — 1. Using both this fact and Little’s 
formula (see p. 318), we may write that the average time that a customer 
entering the system spends in this system is equal to 


N 


eee 6.66 
A(1 — 7) 29) 
For the customers entering the system, we still have that S = 1/p (by 
assumption). Then 
2 N 1 
= ot 6.67 
@ M1l—a) pb eer) 


Remark. If we consider an arbitrary customer arriving in the system, the 
average time that she will spend in it is then J’ = N/A, as noted previously. 
Indeed, in this case the random variable T is of mixed type, and we may write 
that 


E(T] = Elf | T =0]P[T = 0] + E[f | T > O|P[T > 0 
- : 


sia ale 7c Rear 


1l-a™) => (6.68) 
We also have 


(1 — me) (6.69) 
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Example 6.2.4. We consider a queueing system with a single server and finite 
capacity c = 3, in which customers arrive according to a Poisson process with 
rate \ and the service times are independent exponential random variables, 
with parameter yu. When the system is full, the customer who arrived last will 
be served before the one standing in line in front of him. 


The balance equations of the system are the following: 


state 7 departure rate from 7 = arrival rate to 7 


Ato = UT 
(A+ p)my = Amo + pte 
(A + pte = Amy + pT 
pn3 = AT 


wWOnNnre © 


Notice that these equations are the same as the ones obtained in the case of the 
M/M/1/3 system, although the queue discipline is not the default one (that 
is, first-come, first-served). When pz: = A, we know that the solution of these 
equations, under the condition ae nj =1, ism; = 1/4, for j = 0,1, 2,3. 

Suppose that a customer arrives and finds exactly one person in the system. 
We will calculate the mathematical expectation of the total time T; that this 
new customer will spend in the system if uw = A. Let K be the number of 
customers who will arrive after the customer in question but will be served 
before her (if the case may be). By the memoryless property of the exponential 
distribution, we may write that 


K +1 ~ Geom(1/2) 


It follows that 


oo k+1 
E(t, | K =k|P[K =" "= yo ee* (5) 


Me 


E(T,| = 


k=0 k=0 
9 foe) 1 k4+1 1 foe) 1 k4+1 
“5 (a) +3" Gs) 
k=0 k=0 
2 1 ye | 1 3 


Finally, let Tz be the total time that an entering customer will spend in the 
system, and Jet LZ be the number of customers already present in the system 
upon his arrival. We may write that 


E(T] - Sr Bh|L= IP PIL =] 
t=0 
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Given that (in stationary regime) 


Pib==PIN=1N<2]= FS as for | = 0,1,2 


we then deduce from what precedes that 


Too he es ee 
Bin) "=5(5+5 +5) =5 


Notice that when L = 2, it is as if the customer we are interested in had 
arrived at the moment when there was exactly one customer in an M/M/1/3 
system with service policy FIFO (first-in, first-out). 


6.3 Queues with many servers 


6.3.1 The model M/M/s 


An important generalization of the 14/M/1 model is obtained by supposing 
that there are s servers in the system and that they all serve at an exponential 
rate yu. The other basic assumptions that were made in the description of 
the M/M/1 model remain valid. Thus, the customers arrive in the system 
according to a Poisson process with rate ’. The capacity of the system is 
infinite, and the service policy is that by default, namely, first-come, first- 
served. 

We suppose that the arriving customers form a single queue and that the 
customer at the front of the queue advances to the first server who becomes 
available. A system with this waiting discipline is clearly more efficient than 
one in which there is a queue in front of each server, since, in this case, there 
could be one or more idle servers while some customers are waiting in line 
before other servers. 


Remark. A real waiting line in which the customers stand one behind the other 
need not be formed. It suffices that the customers arriving in the system take 
a number, or that the tasks to be accomplished by the servers be numbered 
according to their arrival order in the system. 

Since the customers arrive one at a time and are served one at a time, 
the process {X(t),¢ > 0}, where X(t) represents the number of customers in 
the system at time ¢, is a birth and death process. Note that two arbitrary 
customers cannot leave the system exactly at the same time instant, because 


the service times are continuous random variables. The balance equations of 
the system are the following (see Fig. 6.2 for the M//M/2 model): 


state 7 departure rate from j = arrival rate to j 
0 Ato = UT) 


O0<k<s (A + ky), = (+ 1) peregi +Ame-1 
k>s (A + sp)tE = SUTR41 FATE 
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n nr N 
m 2u 2u 


Fig. 6.2. State-transition diagram for the model M/M/2. 


We can solve this system of equations under the condition }7~°.9 7 = 1. As 
in the case of the M/M/1 model, it is, however, simpler to use Theorem 3.3.4. 


We first calculate the quantities I7,, for k = 1,2,.... The cases when k < s 
and when k > s must be considered separately. We have 
kwifl<k<s 
a Sie 70 
rk { swifk>s am) 
Then 


k — — 


— AXAXKs++ xX 1a 
~ xwxexku ki 


and, fork =s+1,s+2,..., 


AKAK OH KAKAK KA ae eos 
me = (6.72) 
xX QWBwrx-xX sux sux-->x su sisk-e \ 
— 
(k-s) times 


Given that the stochastic process {x (t),t > 0} is irreducible and that 


CO 
BN 
S,:= 0 <0 => pia re (6.73) 
k=1 


we can indeed appeal to Theorem 3.3.4. We first have [see Eq. (3.301)] 


k=0 
s-l f4 s ie ~1 

fae a ig ste | (6.74) 

2 il” sl (=) 

Once 7 has been calculated, we set 
k 
Fm ifk=0,1,...,8 

Tk = : (6.75) 


p* 
Fare fk =s+1,s+2,... 
sis’ 
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We will now calculate, in terms of 7, the average number of waiting 
customers in the system in stationary regime. We have 


fos) fos) ok 
= Yo k-s)m= Yo k-9) jm Smo So (h ~ 3) (2) 
k=s+1 k=s+1 k=s+1 
(6.76) 
Next 
oe) s 
p k p stl . p j-1 p st+1 p -1 1 
S> (r—3) (2) = (2) 5 (2) = - 
ena ( ) ) re 5 (2) e) ( _) 1 — (p/s) 
(6.77) 
from which we obtain 
= pt 62 pe 1 
= ny ane eae 6.78 
Na sls (s— py” sls (l- ee" ee) 
where € := p/s. We can also write that 
: 1 € 
Sys = 6.79 
Na ‘al i get (1 oe aie ( ) 


From Ng and Little’s formula, we deduce all the other quantities of inter- 
est. Since the average entering rate of customers into the system is A, = A, 
we may write that 

= No ; pet ss a 
@ “dst (s— p)? — p)? re £080) 


because S = 1/y for every server (by assumption). 
Finally, we have 


Ns =\AS =p and N=AT=No+t+p (6.81) 


Remarks. i) We can also calculate, in particular, the probability 7 that all 
the servers are busy. We have 


pk 8 p\k 
Th = im = aa Lae hi (£) 
k>s k=s k=s 
PS°S (2)? 4 _ = 6.82 
~ gt ONG T=(p/s) al(e—p) - 82) 


ii) If the number s of servers tends to infinity, then we find that 


k 
tm] —e-/# and mm, — AY Me for k =1,2,... (6.83) 
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That is, in the case of the 1¢/M/oo model, we have 
T, = P|Y =k], where Y ~ Poi(A/p) (6.84) 
Since No = Q = 0 (because there is no waiting time), we then directly obtain 


Vania Cae and T=5=— (6.85) 
Lb 


As we did for the M/M/1 model, we can explicitly find the distribution of 
the random variable Q, namely, the time that an arbitrary customer spends 
waiting in line. This variable is of mixed type. We have 


PQ =0)=1-—m™ (6.86) 
and, for t > 0, 
P0<Q<t=S PO <Q<t|R=r]P[R=r] (6.87) 


where FR is the number of customers in the system upon the arrival of the 
customer of interest. When at least s customers are in the systern, the time 
needed for some customer to depart is a random variable having an exponen- 
tial distribution with parameter sy. It follows that 


Q|{R=r}~G(r—s+1,sp) forr=s,s+1,... (6.88) 


because the customer must wait until r—s-+1 persons ahead of him have left 
the system before starting to be served. Making use of the fact that 


P|R=r)]=7, (6.89) 


(because the arrivals constitute a Poisson process) and proceeding as in the 
case when s = 1, we find that 


P0<Q<tl=m™ (1 = ena) for t > 0 (6.90) 


Finally, adding the probabilities P[(Q = 0] and P[0 < Q < t], we obtain the 
following proposition. 


Proposition 6.3.1. The distribution function of the random variable Q in an 
M/M/s queueing system (for which X < sy) is given by 


Fo(t) = P[Q <t]=1— me! fort >0 (6.91) 
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Remarks. i) With s = 1, we calculate 


-1 
my = (1 a 5) =p (6.92) 
so that 
PIQ <t) 2 1-pe®* fort >0 (6.93) 


which corresponds to the formula (6.27). 
ii) To obtain the distribution of the total time, 7, that an arbitrary customer 
spends in an M/M/s system in equilibrium, we would have to compute the 


convolution product of the probability density functions of the random vari- 
ables Q and S ~ Exp(y). 


Example 6.3.1. The number K of idle servers in an M/M/s system, in sta- 
tionary regime, may be expressed as follows: 


K=s—WNg 


where Ng is the number of customers being served. We then deduce from Eq. 
(6.81) that : 
E|K] =s-—Ns=s-p 


6.3.2 The model M/M/s/c and loss systems 


As we mentioned in Subsection 6.2.2, in reality the capacity c of a waiting 
system is generally finite. In the case of the M/M/s/c model, we can use the 
results on birth and death processes to calculate the limiting probabilities of 
the process. 


Example 6.3.2. Consider the queueing system M/M/2/3 for which A = 2 and 
p. = 4. The balance equations of the system are 


state 7 departure rate from 7 = arrival rate to j 


= 
=) 
= 


0 27% = 4m 
1 (24.4)m 2 2m + (2x 4) 
2 (2+2 x 4)me e 2m, + (2 x 4)a3 
3 (2 x 4)73 8) 27 


We can directly solve this system of linear equations. Equation (0) yields 
Ty = 470. Substituting into (1), we obtain m2 = 47 = 470. Next, we deduce 


from Eq. (3) that 7g = $72 = +7. We can then write that 
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to ++ ed ee 1 = = ae ies See 
Og Der gra ago Weer saa? TA eB gate ER 


Note that this solution also satisfies Eq. (2). 
The average number of customers in the system in equilibrium is given by 


3 
= 16 4 1 27 
ND are g, teh ey tera = ag 


from which the average time that an entering customer spends in the system 
is - 
as x4 
pao _ 27/83 _ 27/538 _ 2 y 50g 
Ae Al—7ms) 2(52/53) 104 
We also find, in particular, that the limiting probability that the system is 


not empty is given by 1 — 79 = 2 


Particular case: The model M/M/s/s 


When the capacity of the system is equal to the number of servers in the 
system, no waiting line is formed. The customers who arrive and find all the 
servers busy do not enter the system and are thus lost. Such a system is called 
a no-watt system or a loss system. 

Let X(t), for ¢ > 0, be the number of customers in the queueing system 
M/M/s/s at time t. We find that the balance equations of the system are 
given by 


state 7 departure rate from j = arrival rate to 7 


0 Aro = BM 
0<k<s (A + ku)ry = (kK +1)pe tepa + ATR-1 
8 SUL Ts = ATs-1 


The birth and death process { X(t), t > 0} is irreducible and, since 


IT, = 


AXAX+++XA _ 1 fA 
xr xk kl 


k 
*) fork =1,2,...,8 (6.94) 
m 
the sum Sj is given by 
s $s 1 d k 
Su SS a (2) (6.95) 
k=l k=1 
This sum is finite for all (positive) values of A and p. Theorem 3.3.4 then 
enables us to write that 


K/h! 
Higa ON a Ei (6.96) 
p/5 
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where p := \/. Thus, the a,’s correspond to the probabilities P[Y = k] of 
a random variable Y having a truncated Poisson distribution. Indeed, if Y ~ 
Poi(p), then we have 


e~ P pk /k! oF /k! 


PIVS RIV So]— SF opi Soe Th 


fork =0,1,...,8 
(6.97) 


Note that in the particular case when \ = p, the limiting probabilities 7, 
do not depend on X (or 2): 
Lu 1/k! 
v0 L/9! 
Moreover, in the general case, the probability zp that all servers are busy 
is simply 


a 


Th = fork =0,1,...,8 (6.98) 


p>/s! 
This formula is known as Erlang’s formula. 


Finally, the average entering rate of customers into the system is A. = 
A(1 — m5). It follows that 


Th =, = for all p > 0 (6.99) 


N=) =X(1- m.)- (6.100) 


Since there is no waiting period in this system, as in the M/M/oo model, we 
have that N = Ns, T =S, and Ng=Q =0. 
Remarks. i) If the number of servers tends to infinity, we obtain 


s 
i jal — oP — orA/b ’ 
fa 2d cP =e (6.101) 
and we retrieve the formula (6.83): 
k 
lim a, = (NEY Mi for k = 0,1,... (6.102) 
s—00 { 


ii) Note that the limiting probabilities may be expressed as follows: 
E(s})* /k! 
jee for k = 0,1,...,8 (6.103) 
dj<o AE[S])” /3! 


Now, it can be shown that the formula (6.103) is also valid for the more general 
model M/G/s/s (known as Erlang’s loss system), in which the service time 
is an arbitrary nonnegative random variable. This result is very interesting, 
because it enables us to treat problems for which the service time does not 
have an exponential distribution. 
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Example 6.3.3. We consider the queueing system M/G/2/2 in which the ar- 
rivals constitute a Poisson process with rate 4 = 2 per hour. 

(a) What is the probability 7 that there are no customers in the system at 
a (large enough) time t if we suppose that the service time S (in hours) is an 
exponential random variable with mean equal to 1/4? What is the average 
number of customers in the system in equilibrium? 

(b) Calculate the value of 79 if S is a continuous random variable whose 
probability density function is 


fs(s) =64se~** for s >0 


Solution. (a) When S ~ Exp(js = 4), we have the M/M/2/2 model with 
p=2/4=1/2. We seek 


1 1 8 
SS Itete 8 
We also find that 
1 1 
1. = 570. = 75 and Ri Gm a 
It follows that - 8 4 ; 6 
NR Nag eX ae ag 


(b) We can check that the density function above is that of a random variable 
having a gamma distribution with parameters a = 2 and A* = 8. It follows 
that E[S] = a/A* = 1/4. Since the mean of S is equal to that of a random 
variable having an Exp(4) distribution, we can conclude that the value of 
mo is the same as that in (a). Actually, we can assert that all the limiting 
probabilities 7, are the same as the 7%’s in (a). Consequently, we have that 
N = 6/13 as well. If we do not recognize the distribution of the service 
time S, then we must calculate the mean E[S] by integrating by parts, or by 
proceeding as follows: 


ee) oO 
ES) c= i: s- 64se~**ds -| 64s7e~ *5ds 
0 0 
auth ft ition = al a) 1 
= = Ye "dt = <1'(3)= >=7 
8 I . co ea 
Remark. The queueing systems M/M/s (= M/M/s/oo) and M/M/s/s are 
the two extreme cases that can be considered. The model M/M/s/c, with 
8 <c< oo, is the one that can most often represent reality well, because there 
is generally some space where potential customers can wait until being served, 
but this space is not infinite. If the capacity of a queueing system is finite, 
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and if this system is part of a network of queues (see the next subsection), 
then we say that the possibility of blocking of the network exists. 

Moreover, we found that for the system M/M1/s to attain a stationary 
regime, the condition A < su must be satisfied. That is, the arrival rate of 
customers in the system must be smaller than the rate at which the customers 
are served when the s servers are busy. Otherwise, the length of the queue 
increases indefinitely. However, in practice, some arriving customers will not 
enter the system if they deem that the queue length is too long. Some may 
also decide to leave the system before having been served if they consider that 
they have already spent too much time waiting in line. 

To make the model M/M/s more realistic, we may therefore suppose that 
the customers are impatient. A first possibility is to arrange things so that 
the probability that an arriving customer decides to stay in the system and 
wait for her turn depends on the queue length upon her arrival. For example, 
in Exercise no. 4, p. 346, we suppose that the probability r, that an arriv- 
ing customer who finds n persons in the system decides to stay is given by 
1/(n +1). This assumption leads to a particularly simple solution. We could 
also suppose that r, = K”, where « € (0, 1], etc. 

When the potential customers decide by themselves not to enter the sys- 
tem, we speak of a priori impatience. We call a posteriori impatience the case 
when the customers, once entered into the system, decide to leave before hav- 
ing been served, or even before their service is completed. This situation may 
be expressed as follows: an arbitrary customer, who entered the system, de- 
cides to leave it if Q > to or if T > t,, where to and ¢, are constants fixed 
in advance. We can also imagine that the time that an arbitrary customer is 
willing to spend in the system (or waiting in line) is a random variable having 
a given distribution (an exponential distribution, for example). 


Example 6.3.4. (a) Suppose that, in the loss system M/M/K/K, for which 
A = ft, the quantity K is a random variable such that 


Kea 1 with probability 1/2 
~ | 2 with probability 1/2 


(i) Calculate the average number of customers in the system in stationary 
regime. 

(ii) For which value of K is the average profit (per time unit) larger if each 
customer pays $x per time unit and each server costs $y per time unit? 


(b) Redo question (i) of part (a) if K is instead a random variable such that 
PIK = k] = 1/3, for k = 1, 2, and 3, and if the capacity of the system is 
c = 3, so that potential customers can wait to be served (when K = 1 or 2). 


Solution. (a) (i) If K = 1, we have (see the formula (6.96) or (6.98)): 


d/ 1 
and — =— 
me T+ 2 


ee eee 
x S 
La 2 
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so that E[N | K = 1] =1/2. When K = 2, we find that 
1 A=p 2 
—_—_,- “St =, 
Mea fd 5 
1+44(A) 3 


It follows that E[N | K = 2] = 4/5. Finally, we have 


1 
To = Ti =7%, and m= 30 


13 


N=E(N]= 5{BIN | K =1]+4 BIN | K =2}} = 20 


(ii) If K = 1, the average profit per time unit, Pr, is given by $($2—y), 
while Pr = $ (2 r-2 y) when K = 2. Therefore, the value K = 1 is the one 
for which the average profit is larger if and only if 


(b) First, if kK = 1 (and c = 3), we have (see the formula (6.49)) mj = 1/4, for 
i1=0,1,2,3. Then ELN | K = 1] = 3/2. Next, if K = 2, we write the balance 
equations of the system (see Ex. 6.3.2): 


state j departure rate from 7 = arrival rate to 7 


Aro ® UT 
(A+ p)m4 el Ato + 2pT2 
(A + 2p) 72 ] Any + 273 
(4) 
2um3 = AT? 


wo nF © 


When A = y, we find that (1) implies that 7 = 71. Furthermore, (2) then 
implies that m2. = 71/2. It follows, from (4), that 3 = 7/4. Making use of 
the condition mp + 7, + m2 + 73 = 1, we obtain that 


4 2 1 
To = Ty il 2 Ti and 73 li [ | 
Finally, with K = 3, we calculate 
i rzn 6 1 1 
T= 37 Tg? 7 = Mor M2 = 570 and 73 = a7 
1+44(4) 4+(3) a 
Be up 2 wp 3! 
so that 


6 3 1 15 
N|K =3)=0x —4+1x—42x543x—=— 
mae. <a5t <p. igre 1G 1 


The average number of customers in the system (in stationary regime) is 


therefore ive ‘e os 
E\N] (5 sored z) 48 
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6.3.3 Networks of queues 


We consider a network made up of k queueing systems. In the ith system, 
there are s; servers (each of them serving only one customer at a time), for 
4 = 1,2,...,k. We suppose that the capacity of each system is infinite and 
that customers, coming from outside the network, arrive in system 7 according 
to a Poisson process with rate 6;. The k Poisson processes are independent. 

After having left the queueing system i, an arbitrary customer goes to 
system j € {1,2,...,k} with probability p;;, so that the probability that the 
customer leaves the network (after having been served in system 7) is given 
by 


k 
Pio :=1-S pj 20 (6.104) 


j=l 


We assume that the probability that a given customer remains indefinitely 
in the network is equal to zero. Finally, the service times are independent 
exponential random variables with rates ;, for i = 1,...,k, and are also 
independent of the times between the successive arrivals. 


Remarks. i) A network of this type is said to be open, because the customers 
can enter and leave the system. We could also consider the case when the 
network is closed, that is, the number of customers is constant, and they 
move indefinitely inside the network. 

ii) Notice that the probability p; ; may be strictly positive, for any i. That is, 
it is possible that a customer, after having departed system i, returns to this 
system immediately. 


Let X(t) := (X1(t),...,X,(t)), where X;(t) designates the number of 
customers in system 7 at time t, for i = 1,...,k. We want to obtain the 
distribution of X(t) in stationary regime. Let A; be the total rate at which 
customers arrive in system j. Since the arrival rate into a queueing system 
must be equal to the departure rate from this system, the A;’s are the solution 
of the system of equations 


k 
A= Oj; +> Ai pig for j=1,2,...,k (6.105) 
i=1 


Once this system has been solved, the next theorem, known as Jackson’s® 


theorem, gives us the solution to our problem. 


Theorem 6.3.1. Let N; := limtoo Xi(t). If A; < sipj, fori =1,...,k, then 


k 
Jim P[X(t) = n] = P ne = va (6.106) 
t=1 


3 James R. Jackson, emeritus professor at UCLA (University of California, Los 
Angeles), in the United States. 
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where n := (n1,...,nx), and N; is the number of customers in an M/M/s; 
queueing system in stationary regime. 


Remarks. i) Under the assumption that all customers eventually leave the 
network, it can be shown that the system (6.105) has a unique solution. 


ii) The statement of the theorem is surprising, because it implies that the 
random variables N; are independent. If an arbitrary customer cannot return 
to a system he already departed, then the arrivals in each system constitute 
Poisson processes, because the departure process of an M/M/s queue is (in 
stationary regime) a Poisson process with rate 2 (if X < sj). Moreover, these 
Poisson processes are independent. In this case, the result of the theorem is 
easily proved. However, when a customer may be in the same system more 
than once, it can be shown that the arrival processes are no longer Pois- 
son processes. Indeed, then the increments of these processes are no longer 
independent. Now, according to the theorem, the random variables are nev- 
ertheless independent. 


iii) When there is a single server per system, so that s; = 1, the network 
described above is called a Jackson network. We then deduce from the formula 
(6.11) that 


Jim P[(Xi(t),.-- X(t) = (mi,--- -me)] = I (2) (1 zs **) (6.107) 


(if A; < pu, for all i). The average number of customers in the network in 
equilibrium is then given by 


N= N= yo (6.108) 


i=1 i=1 


Finally, given that the average entering rate of customers (coming from the 
outside) into the network is 


k 
de = 04; (6.109) 


pace (6.110) 


Example 6.3.5. The simplest example of a Jackson network is that of a sequen- 
tial system in which there are two servers and the arriving customers must 
necessarily go the first server, and next directly to the second one. Then they 
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leave the system. We suppose that the first queueing system is an M/M/1 
model, with service rate u;. Thus, the arrivals in the network constitute a 
Poisson process with rate A. Similarly, we suppose that the capacity of the 
second queueing system is infinite and that the server performs the desired 
service (to only one customer at a time) at an exponential rate jue. Finally, 
all the service times and the times between the successive arrivals are inde- 
pendent random variables. 

Since the customers cannot find themselves more than once in front of the 
same server, we can assert, if \ < 41, that the departure process of the first 
system (in stationary regime) is also a Poisson process with rate A, so that 
the second queueing system is an M/M/1 model as well. We deduce from 
Jackson’s theorem that 


Tnayna *= jim P{(X1(t), Xo(t)) = (m1, n2)] 


SODA CA om 


for ny and ng € {0,1,...}. 

We can show that the formula above is valid by checking that the joint 
limiting probabilities 7), ,n, satisfy the balance equations of the network. Sup- 
pose, to simplify further, that 4; = pa := ps. These balance equations are then 


state (i, 7) departure rate from (i, 7) = arrival rate to (i, 7) 


(0,0) AT0,0 = UT0,1 
(n1,0),n1 > 0 (A + 2)Tn,,0 =HTn,,1 + ATn,~1,0 
(0,n2),n2 > 0 (A + 2)t0,np =E(Tojn2+1 + Tiyn2-1) 
(ni, 72), NiNe >0 (A + 21) Tn, ,n2 = (Tr, no+1 + Teil) 
+ATn,=-1,n2 


When py = 2 = LM, we can rewrite the formula (6.111) as follows: 


Ny+n2 2 
re (2) (1 ‘ *) (6.112) 
LU Ll 


We have, in particular, 


(A ot, 24) Tny no a M(%ny,n241 ef Tny+1,n2-1) a ATn,~1,n2 
=— 


Ny+tne nytnetl nytn2 ny—-l+ne2 
(A + 2) (2) =" (2) + (2) | +2 (2) 
a bt LL lt 
— 
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By the uniqueness of the solution, under the condition 7,4, 9) Trine = 1, 
we can then conclude that the probabilities 7,n, are indeed given by the 
formula (6.111). 


Remarks. i) When py = fe = p, the limiting probabilities 7,,,, depend 
only on the sum n; + m2. However, the probability n,n, is not equal to the 
probability that there will be exactly n,; + n2 customers in the network in 
stationary regime. Indeed, we calculate, for example, 


Jim P[Xi(t) + X2(t) = 1] pe 2 lim P[X1(t) = 1, Xo(t) = 0} 


2 
= 2 (>) (1 - =| (6.114) 
Lb Lb 
Moreover, we have 


Ee £0" 


nitn2=0 1tn2= 
2 
=(1 *) ea a4 (6.115) 
pe 1— rr pe 
while 
> jim P[X1(t) + Xo(t) =m + ng] = 1 (6.116) 
nitn2=0 


ii) We can also write that 


Tnyng = lim P[X1(t) = ni +12, Xo(t) = 0] 


= Jim P[X,(t) = 0, Xo(t) = my + ng] (6.117) 
More generally, we have 
Tang = dim P[Xi(t) = 4, Xe(t) = J] (6.118) 


for all nonnegative integers 7 and j such that i+ j = ny + ng. Thus, when 
/4) = pa, all the possible distributions of the n1 + m2 customers between the 
two servers are equally likely. 


iii) Finally, the network described in this example is different from the M/M/2 
model, even if the two service rates are equal. Indeed, when there are exactly 
two customers in this network, both these customers may stand in front of 
server 1 (or server 2), whereas, in the case of the M/M//2 model, there must 
be one customer in front of each server. 
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6.4 Exercises 


Remark. In the following exercises, we assume that the service times are ran- 
dom variables that are independent among themselves and are independent 
of the times between successive arrivals. 


Section 6.2 


Question no. 1 
Calculate the average number of arrivals in the system during the service 
period of an arbitrary customer for the queueing model M/M/1/3. 


Question no. 2 

Drivers stop to fill up their cars at a service station according to a Poisson 
process with rate \ = 15 per hour. The service station has only one gasoline 
pump, and there is room for only two waiting cars. We suppose that the 
average service time is equal to two minutes. 


(a) Calculate N and Ng. Why is N #4 Ng +1? 


(b) If we suppose that an arriving driver who finds the three spaces occupied 
will go to another service station, what proportion of potential customers is 
lost? 


Question no. 3 

Airplanes arrive at an airport having a single runway according to a Pois- 
son process with rate \ = 18 per hour. The time during which the runway is 
used by a landing airplane has an exponential distribution with mean equal 
to two minutes (from the moment it receives the landing authorization). 


(a) Knowing that there is at most one airplane in the system (at a given time 
instant), what is the probability that an arriving airplane will have to wait 
before being allowed to land? 


(b) Given that an airplane has been waiting for the authorization to land for 
the last 5 minutes, what is the probability that it will have landed and cleared 
the runway in the next 10 minutes? 


Question no. 4 

We suppose that the probability that an arriving customer in an M /M/\ 
queueing system decides to stay and wait until being served is given by 
1/(n + 1), where n is the number of customers in the system at the time 
when the customer in question arrives, for n =0,1,2,.... 
(a) Calculate N and Ng. 


(b) What is the percentage of customers who decide not to enter the system? 


Question no. 5 

We wish to compare two maintenance policies for the airplanes of a certain 
airline company. In the case of policy A (respectively, B), the airplanes arrive 
to the maintenance shop according to a Poisson process with rate A4 = 1 
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(resp., Ag = 1/4) per day. Moreover, when policy A (resp., B) is used, the 
service time (in days) is an exponential random variable with parameter fu4 = 
2 (resp., the sum of four independent exponential random variables, each of 
them with parameter j1g = 2). In both cases, maintenance work is performed 
on only one airplane at a time. 


(a) What is the better policy? Justify your answer by calculating the average 
number of airplanes in the maintenance shop (in stationary regime) in each 
case. 


Indication. The average number of customers in a queueing system M/G/1 
(after a long enough time) is given by 


NBS? 


N= EIS] + oa Rsh 


where S is the service time and 4 is the average arrival rate of customers. 


(b) Let N be the number of airplanes in the maintenance shop in stationary 
regime. Calculate the distribution of N if policy A is used, given that there 
are two or three airplanes in the shop (at a particular time instant). 


(c) If policy A is used, what is the average time that an airplane, which has 
already been in the maintenance shop for two days, will spend in the shop 
overall? 


Question no. 6 

We consider a queueing system in which there are two types of customers, 
both types arriving according to a Poisson process with rate A. The customers 
of type I always enter the system. However, the type II customers only en- 
ter the system if there is no more than one customer in the system when 
they arrive. There is a single server and the service time has an exponential 
distribution with parameter . 


(a) Write the balance equations of the system. 


(b) Calculate the limiting probability that an arbitrary type IJ customer enters 
the system if \ = 1 and wu = 2. 


Indication. The system considered is a birth and death process. 


(c) Calculate the average time that a given arriving customer of type II will 
spend in the system if \ = 1 and w= 2. 


Question no. 7 

We consider a waiting system with a single server and finite capacity c = 
3, in which the customers arrive according to a Poisson process with rate 
A and the service times are independent exponential random variables with 
parameter 4 = 2A. When the system is full, a fair coin is tossed to determine 
whether the second or third customer will be the next one to be served. 


(a) Write the balance equations for this system, and calculate the limiting 
probabilities 7;. 
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(b) Calculate the average time that an arriving customer who finds (exactly) 
one customer in the system will spend in it. 


(c) Calculate the average time that a customer who enters the system will 
spend in it. 


Question no. 8 

Customers arrive at a service facility according to a Poisson process with 
rate \ = 10 per hour. The (only) server is able to serve up to three customers 
at a time. The service time (in hours) has an exponential distribution with 
parameter p = 5, regardless of the number of customers (namely 1, 2, or 3) 
being served at the same time. However, an arbitrary customer is not served 
immediately if the server is busy upon her arrival. Moreover, we suppose that 
the service times are independent random variables and that there can be at 
most three customers waiting at any time. We define the states 


0’: nobody is being served 
0: the server is busy; nobody is waiting 
n: there is (are) n customer(s) waiting, for n = 1,2,3 


(a) Write the balance equations of the system. 
(b) Calculate the limiting probabilities 7,, for all states j. 


(c) What is the probability that an arriving customer will be served alone? 


Question no. 9 (Modification of the preceding question) 

Customers arrive at a service facility according to a Poisson process with 
rate \. The (only) server is able to serve up to two customers at a time. 
However, an arbitrary customer is not served immediately if the server is 
busy upon his arrival. The service time has an exponential distribution with 
parameter yz; when the service is provided to i customer(s) at a time, for 
t = 1,2. Moreover, we suppose that the service times are independent random 
variables and that there can be at most two customers waiting at any time. 
We define the states 


0: nobody is being served 
n;: there is (are) n customer(s) waiting and 7 customer(s) being served 


for n = 0,1,2 and i = 1,2. 
(a) Write the balance equations of the system. Do not solve them. 


(b) Calculate, in terms of the limiting probabilities, the probability that 
(i) the server is serving two customers at a time, given that he is busy, 
(ii) an arriving customer who enters the system will not be served alone. 


Question no. 10 

Customers arrive into a queueing system according to a Poisson process 
with rate 4. There is a single server, who cannot serve more than one customer 
at a time. However, the larger the number of customers in the system is, the 
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faster the server works. More precisely, we suppose that the service time has 
an exponential distribution with parameter u(k) = ae when there are k 
customers in the system, for & = 1,2,.... Moreover, we suppose that the 
service times are independent random variables and that A = 1 and p = 2. 


(a) Calculate the limiting probabilities 7,, for n =0,1,.... 


(b) Let X(t) be the number of customers in the system at time t, for t > 0. 
Calculate E[.X(t) | X(t) < 2] when the system is in equilibrium. 


(c) Let To be the time spent in the system by a customer who arrived, at time 
to, while the system was empty. Calculate the expected value of To, given that 
the following customer arrived at time tg + 1 and the customer in question 
had already left the system. 


Question no. 11 

Drivers arrive according to a Poisson process with rate A to fill up their 
cars at a service station where there are two employees who serve at the 
exponential rates j4; and jo, respectively. However, only one employee works 
at a time serving gasoline. Moreover, there is space for only one waiting car. 
We suppose that 


e when the system is empty and a customer arrives, employee no. 1 fills up 
the car, 

e when employee no. 1 (respectively, no. 2) finishes filling up a car and 
another car is waiting, there is a probability equal to p; (resp., p2) that 
this employee will service the customer waiting to be served, independently 
from one time to another. 


Finally, we suppose that the service times are independent random vari- 
ables. 


(a) Let X(t) be the state of the system at time t. Define a state space in such 
a way that the stochastic process {X(t),t > 0} is a continuous-time Markov 
chain. 


(b) Write the balance equations of the process. 


(c) Calculate, in terms of the limiting probabilities, the probability that 

(i) an arbitrary customer entering the system will be served by employee 
no. 2, 

(ii) two customers arriving consecutively will be served by different em- 
ployees, given that the first of these customers arrived while there was exactly 
one car, being filled up by employee no. 1, in the system. 


Question no. 12 

We consider the queueing system M/M/1. However, the customers do not 
have to wait, because the server is able to serve all the customers at one time, 
at an exponential rate jy, regardless of the number of customers being served. 
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Calculate 

(a) the limiting probability, z,, that there are n customers being served, for 
n=0,1,..., 

(b) the variance of the number of customers being served when the system is 
in equilibrium. 


Question no. 13 

Suppose that customers arrive at a service facility with a single server 
according to a Poisson process with rate \. The server waits until there are 
four customers in the system before beginning to serve them, all at once. 
The service times are independent random variables, all having a uniform 
distribution on the interval (0,1). Moreover, the system capacity is equal to 
four customers. What fraction of time, 7;, are there i customer(s) in the 
system, over a long period? 


Indication. Use the results on renewal processes. 


Question no. 14 

Suppose that the times between the arrivals of consecutive customers in 
a certain queueing system are independent random variables uniformly dis- 
tributed over the interval (0,1). The service time is exponentially distributed, 
with parameter jy. Finally, the (only) server is able to serve all the customers 
at once, so that there is no waiting. Calculate the limiting probability that 
the server is busy. 


Indication. Use the results on renewal processes. 


Question no. 15 

We modify the M/M/1/4 queueing system as follows: the server always 
waits until there are at least two customers in the system before serving them, 
two at a time, at an exponential rate pu. 
(a) Write the balance equations of the system. 
(b) Calculate the limiting probabilities, 7,, for n = 0,1,2,3,4, in the case 
when A = wt, where \ is the average arrival rate of the customers. 
(c) With the help of the limiting probabilities calculated in (b), find 

(i) the probability that the system is not empty at the moment when the 
server has just finished serving two customers, 

(ii) the variance of the number X(t) of customers in the system (in equi- 
librium) at time t, given that X(t) < 2. 


Section 6.3 


Question no. 16 

Customers arrive according to a Poisson process with rate A at a bank 
where two clerks work. Clerk 1 (respectively, 2) serves at an exponential rate 
41 (resp., j42). We suppose that the customers form a single queue and that, 
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when the system is empty, an arriving customer will go to clerk 1 (resp., 2) 
with probability p; (resp., 1 — p,). On the other hand, when a customer must 
wait, she will eventually be served by the first available clerk. We also suppose 
that an arbitrary customer can enter the bank only if there are no more than 
10 customers waiting in line. We say that the system is in state n = 0,2,... 
if there are m customers in the bank, and in state 1, (resp., lg) if there is 
exactly one customer in the bank and if this customer is being served by clerk 
1 (resp., 2). 

(a) Write the balance equations of the system. 


(b) In terms of the limiting probabilities, what is the probability that an 
entering customer will be served by clerk 1? 


Question no. 17 

In a certain garage, there are three mechanics per work shift of eight hours. 
The garage is open 24 hours a day. Customers arrive according to a Poisson 
process with rate \ = 2.5 per hour. The time a mechanic takes to perform 
an arbitrary task is an exponential random variable with mean equal to 30 
minutes. 


(a) What proportion of time are all the mechanics busy? 
(b) How much time, on average, must a customer wait for his car to be ready? 


Question no. 18 

In a small train station, there are two counters where the travelers can buy 
their tickets, but the customers form a single waiting line. During the slack 
hours, only one counter is manned continuously by a clerk. When there is at 
least one customer waiting to be served, the second clerk opens his counter. 
When this second clerk finishes serving a customer and there is nobody wait- 
ing, he goes back to attending to other tasks. We suppose that the clerks both 
serve in a random time having an exponential distribution with parameter jz 
and that, during the slack hours, the customers arrive according to a Poisson 
process with rate 4. We also suppose that the slack period lasts long enough 
for the process to reach a stationary regime. The state X(t) of the system 
is defined as being the total number of customers present in the system at 
time ¢. 


(a) Calculate the limiting probabilities of the process {X(t),t > 0} if A < 2y. 


(b) Write the balance equation for the state 12 corresponding to the case when 
only the second clerk is busy (serving a customer). 


(c) What fraction of time is the second counter open? 


Question no. 19 

Customers arrive at a hairdresser’s salon according to a Poisson process 
with rate \ = 8 per hour. There are two chairs, and the two hairdressers’ ser- 
vice times are exponential random variables with means equal to 15 minutes. 
Moreover, currently, there is no room where potential customers could wait 
for their turn to have their hair cut. 
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(a) The owner considers the possibility of enlarging the salon, so that she could 
install an additional chair and hire a third hairdresser. This, would increase 
her operation costs by $20 per hour. If each customer pays $10, would the 
enlargement be profitable? Justify. 


Indication. Calculate the average rate at which customers enter the salon. 


(b) Another possibility the owner considered consists of enlarging the salon to 
install a chair where one potential customer could wait to be served. In this 
case, the increase in operation costs would be equal to $5 per hour. Would 
this possibility be profitable? Justify. 


Question no. 20 
We consider the queueing system M/A1/2/3 (see Example 6.3.2). 


(a) Write the balance equations of the system, and calculate the limiting 
probabilities 7; if A = 2. 

(b) Let T* be the total time that an entering customer will spend in the 
system. Calculate the expected value of T* if y = 1. 


(c) Suppose that the customers form two waiting lines, by standing at random 
in front of either server. Calculate, with ~ = 1, the average time that an 
arbitrary customer who enters the system, and finds two customers already 
present, will spend in this system if we assume that the number of customers 
in each queue is then a random variable having a binomial distribution with 
parameters n = 2 and p = 1/2. 


Question no. 21 

We consider a queueing system with two servers. The customers arrive 
according to a Poisson process with rate \ = 1, and the system capacity is 
equal to four customers. The service times are independent random variables 
having an exponential distribution. Each server is able to serve two customers 
at a time. If a server attends to only one customer, he does so at rate pp = 2, 
whereas the service rate is equal to 1 when two customers are served at the 
same time. 


Indication. If two customers are served together, then they will leave the 
system at the same time. Moreover, if there are two customers in the system, 
then one of the servers may be free. 


(a) Write the balance equations of the system. 


(b) Let T* be the total time that a given customer entering the system will 
spend in it. 

(i) Calculate, in terms of the limiting probabilities, and supposing that no 
customers arrive during the service period of the customer in question, the 
distribution function of T*. 

(ii) Under the same assumption as in (i), does the random variable T* 
have the memoryless property? Justify. 
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Question no. 22 

Customers arrive according to a Poisson process with rate A = 5 per hour 
in a system with two servers. The probability that an arbitrary customer 
goes to server no. 1 (respectively, no. 2) is equal to 3/4 (resp., 1/4). The 
service times (in hours) are independent exponential random variables with 
parameters 4; = 6 and po = 4, respectively. A customer who goes to server 
no. 2 immediately leaves the system after having been served. On the other 
hand, after having been served by server no. 1, a customer (independently 
from one time to another) 


leaves the system with probability 1/2 
goes to server no. 2 with probability 2/5 
returns in front of server no. 1 with probability 1/10 


Moreover, there is no limit on the number of customers who can be in the 
system at any time. 

Let (n,m) be the state of the system when there are n customers in front 
of server no. 1 and m customers in front of server no. 2. 


(a) Calculate t(_m), for all n,m > 0. 


(b) Calculate the average number of customers in the system at a large enough 
time instant, given that the system is not empty at the time in question. 


(c) What is the average time that an arbitrary customer who arrives in the 
system and goes to server no. 1 will spend being served by this server before 
leaving the system if we suppose that the customer in question never goes to 
server no. 2? 


Question no. 23 

Customers arrive according to a Poisson process with rate outside a bank 
where there are two automated teller machines (ATM). The two ATMs are 
not identical. We estimate that 30% of the customers use only ATM no. 1, 
while 20% of the customers use only ATM no. 2. The other customers (50%) 
make use of either ATM indifferently. The service times at each ATM are 
independent exponential random variables with parameter p. Finally, there is 
space for a single waiting customer. We define the states 


0: the system is empty 
(n,m): there are m customers for ATM no. 1 and m customers 
for ATM no. 2, forl1<n+m<2 
3: the system is full 


(a) Write the balance equations of the system. Do not solve them. 


(b) Calculate, in terms of the limiting probabilities, 

(i) the variance of the number of customers who are waiting to use an 
ATM, 

(ii) the average time that an arbitrary customer, who enters the system 
and wishes to use ATM no. 2, will spend in the system. 
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Question no. 24 

We consider a queueing system in which ordinary customers arrive accord- 
ing to a Poisson process with rate \ and are served in a random time having 
an exponential distribution, with parameter ps, by either of two servers. Fur- 
thermore, there is a special customer who, when she arrives in the system, is 
immediately served by server no. 1, at an exponential rate ,. If an ordinary 
customer is being served by server no. 1 when the special customer arrives, 
then this customer is returned to the head of the queue. We suppose that 
the service times are independent random variables and that the special cus- 
tomer spends an exponential time (independent of the service times), with 
parameter 4,, outside the system between two consecutive visits. 


(a) Suppose that if an arbitrary customer is returned to the queue, then he 
will resume being served as soon as either server becomes available. Define an 
appropriate state space, and write the balance equations of the system. 


(b) Suppose that the system capacity is c = 2, but that if a customer is 
displaced by the special customer, then she will wait, a few steps behind, until 
server no. 1 becomes available to resume being served by this server (whether 
server no. 2 is free or not). Define a state space such that the stochastic process 
{X(t),t > 0}, where X(t) represents the state of the system at time f, is a 
continuous-time Markov chain. 


(c) Suppose that the system capacity is c = 2 and that, if a customer is dis- 
placed by the special customer, then he will go to server no. 2 only if this 
server is free upon the arrival of the special customer in the system. Other- 
wise, he will wait, a few steps behind, before server no. 1 becomes available. 
Let K be the number of times that a given customer, who has started re- 
ceiving service from server no. 1, will be displaced by the special customer. 
Calculate P[K = 1] in terms of the limiting probabilities of the system (with 
an appropriate state space). 


Question no. 25 
Let N be the number of customers in an M/G/2/2 (loss) system after a 
time long enough for the system to be in stationary regime. 


(a) Calculate V[N | N > 0] if the service time, S, has a uniform distribution 
on the interval (0, 1) and if the average arrival rate of customers in the system 
is \ = 4. 

(b) Calculate V[N | X = 1/4] if S has an exponential distribution with pa- 
rameter 1/X, where X ~ U(0,1) and \ = 2. 


Question no. 26 

Suppose that we modify the M//M/2 queueing system as follows: when a 
server is free, he assists (if needed) the other server, so that the service time, S, 
has an exponential distribution with parameter 2. If a new customer arrives 
while a customer is being served by the two servers at the same time, then 
one the servers starts serving the new customer. 
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(a) Calculate the limiting probabilities if we suppose that A < 2u. 


(b) Suppose that the system capacity is c = 2 and that A = p. Calculate the 
average number of customers in the system in stationary regime, given that 
the system is not full. 


Question no. 27 

Let X(t) be the number of customers at time t > 0 in a queueing sys- 
tem with s servers and finite capacity c, for which the time 7 between two 
consecutive arrivals is a random variable such that f,-(t) = 2te*, for t > 0. 
We assume that the times between the arrivals of customers are independent 
and identically distributed random variables. Similarly, the service times are 
independent random variables having, for each server, the same probability 
density function as r. We define the stochastic process {Y(t),t > 0} by 


Y(t) = X(g(t)) fort >0 


where g(t) is a one-to-one function of t. Find a function g such that {Y(t), 
t > 0} is an M/M/s queueing system with \ = 4 = 1 (and finite capacity c). 
Justify. 


Question no. 28 

We consider the loss system M//G/2/2. Suppose that the service times 
have independent exponential distributions with parameter O, where O is a 
random variable such that 


fo(@) = 5 (26? + 20 + 1) for0<6@<1 


Suppose also that the average arrival rate of customers is \ = 1. Calculate, 
assuming they exist, the limiting probabilities z;, for 7 = 0,1, 2. 


Question no. 29 

In the M/M/2 queueing system, we define the random variable S as being 
the first time that both servers are busy. Let 

F = exactly two customers arriving in the interval (0, ¢]. 
Calculate P[S < t | F']. 


Question no. 30 

A hairdresser and her assistant operate a salon. There are two types of 
customers: those of type I prefer to have their hair cut by the hairdresser but 
are willing to be served by her assistant, while those of type II want to be 
served by the assistant only. The type I (respectively, type II) customers arrive 
at the salon according to a Poisson process with rate A; (resp., Az). Moreover, 
the two Poisson processes are independent. Finally, the hairdresser (resp., 
the assistant) serves in a random time having an exponential distribution 
with parameter jz; (resp., 2), and the service times are independent random 
variables. Answer the following questions, supposing that there is no room 
where potential customers can wait until being served: 
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(a) define a state space that enables you to answer part (c); 
(b) write the balance equations; 
(c) give, in terms of the limiting probabilities, 

(i) the average number of customers in the system; 


(ii) the average time that an arbitrary entering customer will spend in the 
system. 


Question no. 31 
Redo the preceding question, supposing that the system capacity is infinite 
and that 


a) only the potential type I customers are willing to wait until being served 
those of type II go away if the assistant is busy), 


( 
( 
(b) only the potential type II customers are willing to wait until being served 
(those of type I go away if the hairdresser and her assistant are busy), 

( 


c) all the potential customers are willing to wait until being served. 


Question no. 32 

Suppose that in the queueing system M/M/2/c, with c = 3, server no. 1 
serves only one person at a time, at rate ,, while server no. 2 can serve one 
or two persons at a time, from any time instant, at rate j12. Moreover, when 
server no. 1 is free, an arriving customer will go to this server. 


(a) Let X(t) be the number of persons in the system at time t. Define a state 
space such that the stochastic process {X(t),t > 0} is a continuous-time 
Markov chain. 


Remark. Server no. 1 may be free while server no. 2 serves two customers at a 
time. That is, the two customers finish their service period with server no. 2. 
(b) Write the balance equations of the system. Do not solve them. 


(c) In terms of the limiting probabilities, what fraction of time does server 
no. 2 serve two customers at a time, given that she is busy? 


Question no. 33 

Redo the preceding question, supposing that the system capacity is instead 
c = 4 and that server no. 2 can serve one or two persons at a time (at rate 
2) but only from the same time instant. 


Appendix A: Statistical Tables 


A.1: Distribution Function of the Binomial Distribution 
A.2: Distribution Function of the Poisson Distribution 
A.3: Distribution Function of the N(0,1) Distribution 
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Table A.1. Distribution Function of the Binomial Distribution 


P 
0.05 0.10 0.20 0.25 0.40 0.50 


0.6400 0.5625 
0.9600 0.9375 


0.5120 0.4219 
0.8960 0.8438 
0.9920 0.9844 


0.4096 0.3164 
0.8192 0.7383 
0.9728 0.9493 
0.9984 0.9961 


0.3277 0.2373 
0.7373 0.6328 
0.9421 0.8965 
0.9933 0.9844 
0.9997 0.9990 


0.1074 0.0563 
0.3758 0.2440 
0.6778 0.5256 
0.8791 0.7759 
0.9672 0.9219 
0.9936 0.9803 
0.9991 0.9965 
0.9999 0.9996 
1.0000 1.0000 


0.0352 0.0134 
0.1671 0.0802 
0.3980 0.2361 
0.6482 0.4613 
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Table A.1. Continued 


0.05 


0.10 


WON Oo fF wWNY KF © 


— 
re © 


0.9994 
0.9999 
1.0000 


0.3585 
0.7358 
0.9245 
0.9841 
0.9974 
0.9997 
1.0000 


0.9873 
0.9977 
0.9997 
1.0000 


0.1216 
0.3917 
0.6769 
0.8670 
0.9568 
0.9887 
0.9976 
0.9996 
0.9999 
1.0000 


0.20 
0.8358 
0.9389 
0.9819 
0.9958 
0.9992 
0.9999 
1.0000 


0.0115 
0.0692 
0.2061 
0.4114 
0.6296 
0.8042 
0.9133 
0.9679 
0.9900 
0.9974 
0.9994 
0.9999 
1.0000 


P 
0.25 


0.6865 
0.8516 
0.9434 
0.9827 
0.9958 
0.9992 
0.9999 
1.0000 


0.0032 
0.0243 
0.0913 
0.2252 
0.4148 
0.6172 
0.7858 
0.8982 
0.9591 
0.9861 
0.9961 
0.9991 
0.9998 
1.0000 


0.40 
0.2173 
0.4032 
0.6098 
0.7869 
0.9050 
0.9662 
0.9907 
0.9981 
0.9997 
1.0000 


0.0000 
0.0005 
0.0036 
0.0160 
0.0510 
0.1256 
0.2500 
0.4159 
0.5956 
0.7553 
0.8725 
0.9435 
0.9790 
0.9935 
0.9984 
0.9997 
1.0000 


0.50 
0.0592 
0.1509 
0.3036 
0.5000 
0.6964 
0.8491 
0.9408 
0.9824 
0.9963 
0.9995 
1.0000 


0.0000 
0.0002 
0.0013 
0.0059 
0.0207 
0.0577 
0.1316 
0.2517 
0.4119 
0.5881 
0.7483 
0.8684 
0.9423 
0.9793 
0.9941 
0.9987 
0.9998 
1.0000 
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Table A.2. Distribution Function of the Poisson Distribution 


x 
0 
1 
2 
3 
4 
5 
6 
7 
8 
9 
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Table A.3. Distribution Function of the N(0, 1) Distribution 


+0.06 +0.07 
0.5000 0.5040 0.5080 0.5120 0.5160 0.5199 0.5239 0.5279 0.5319 0.5359 
0.5398 0.5438 0.5478 0.5517 0.5557 0.5596 0.5636 0.5675 0.5714 0.5753 
0.5793 0.5832 0.5871 0.5910 0.5948 0.5987 0.6026 0.6064 0.6103 0.6141 
0.6179 0.6217 0.6255 0.6293 0.6331 0.6368 0.6406 0.6443 0.6480 0.6517 
0.6554 0.6591 0.6628 0.6664 0.6700 0.6736 0.6772 0.6808 0.6844 0.6879 


0.6915 0.6950 0.6985 0.7019 0.7054 0.7088 0.7123 0.7157 0.7190 0.7224 
0.7257 0.7291 0.7324 0.7357 0.7389 0.7422 0.7454 0.7486 0.7517 0.7549 
0.7580 0.7611 0.7642 0.7673 0.7704 0.7734 0.7764 0.7794 0.7823 0.7852 
0.7881 0.7910 0.7939 0.7967 0.7995 0.8023 0.8051 0.8078 0.8106 0.8133 
0.8159 0.8186 0.8212 0.8238 0.8264 0.8289 0.8315 0.8340 0.8365 0.8389 


0.8413 0.8438 0.8461 0.8485 0.8508 0.8531 0.8554 0.8577 0.8599 0.8621 
0.8643 0.8665 0.8686 0.8708 0.8729 0.8749 0.8770 0.8790 0.8810 0.8830 
0.8849 0.8869 0.8888 0.8907 0.8925 0.8944 0.8962 0.8980 0.8997 0.9015 
0.9032 0.9049 0.9066 0.9082 0.9099 0.9115 0.9131 0.9147 0.9162 0.9177 
0.9192 0.9207 0.9222 0.9236 0.9251 0.9265 0.9279 0.9292 0.9306 0.9319 


0.9332 0.9345 0.9357 0.9370 0.9382 0.9394 0.9406 0.9418 0.9429 0.9441 
0.9452 0.9463 0.9474 0.9484 0.9495 0.9505 0.9515 0.9525 0.9535 0.9545 
0.9554 0.9564 0.9573 0.9582 0.9591 0.9599 0.9608 0.9616 0.9625 0.9633 
0.9641 0.9649 0.9656 0.9664 0.9671 0.9678 0.9686 0.9693 0.9699 0.9706 
0.9713 0.9719 0.9726 0.9732 0.9738 0.9744 0.9750 0.9756 0.9761 0.9767 


0.9772 0.9778 0.9783 0.9788 0.9793 0.9798 0.9803 0.9808 0.9812 0.9817 
0.9821 0.9826 0.9830 0.9834 0.9838 0.9842 0.9846 0.9850 0.9854 0.9857 
0.9861 0.9864 0.9868 0.9871 0.9875 0.9878 0.9881 0.9884 0.9887 0.9890 
0.9893 0.9896 0.9898 0.9901 0.9904 0.9906 0.9909 0.9911 0.9913 0.9916 
0.9918 0.9920 0.9922 0.9925 0.9927 0.9929 0.9931 0.9932 0.9934 0.9936 


0.9938 0.9940 0.9941 0.9943 0.9945 0.9946 0.9948 0.9949 0.9951 0.9952 
0.9953 0.9955 0.9956 0.9957 0.9959 0.9960 0.9961 0.9962 0.9963 0.9964 
0.9965 0.9966 0.9967 0.9968 0.9969 0.9970 0.9971 0.9972 0.9973 0.9974 
0.9974 0.9975 0.9976 0.9977 0.9977 0.9978 0.9979 0.9979 0.9980 0.9981 
0.9981 0.9982 0.9982 0.9985 0.9985 0.9986 0.9986 
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Table A.3. Continued 


+0.00 +0.01 +0.02 +0.03 +0.04 +0.05 +0.06 +0.07 -+0.08 -+0.09 


0.9987 0.9987 0.9987 0.9988 0.9988 0.9989 0.9989 0.9989 0.9990 0.9990 
0.9990 0.9991 0.9991 0.9991 0.9992 0.9992 0.9992 0.9992 0.9993 0.9993 
0.9993 0.9993 0.9994 0.9994 0.9994 0.9994 0.9994 0.9995 0.9995 0.9995 
0.9995 0.9995 0.9995 0.9996 0.9996 0.9996 0.9996 0.9996 0.9996 0.9997 
0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9998 


0.9998 0.9998 0.9998 0.9998 0.9998 0.9998 0.9998 0.9998 0.9998 0.9998 
0.9998 0.9998 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 
0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 
0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 0.9999 
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 


Appendix B: Answers to Even-Numbered 
Exercises 


Chapter 1 
2. (a) 0.0062; (b) ~0.80. 
4, 1~ 1/8 (0.6073). 
6. 379/2187 (~0.1733). 
8. 244/495 (~0.4929). 
10. 1/3. 
12. 
2 c 
ce e—(1+c) 
16. (a) Y~ ae 1}; (b) g(x) = 2F x(x) +1 
18. (a) 2/3; (b) 1/3; (c) 8/3. 
20. (a) 0.3591; (b) ~0.3769; (c) 0.1750. 
22. (a) ~0.4354; (b ) an 80; (c) £81.5%. 
24. (b) (i) 0.1837; ( 7/2. 
26. (a) 
F¥i,¥o(Y1s Y2) = Feg7? {- 5, 302 (yz + 2y3 — 6yry2 — Qype + 2p" »)} 
for all (y1, y2) € R?; (b) 307. 
32. (b) VLX]/2. 
34. fzix(z | «) = fr(z-2). 
36. (a) 


(Soest ax) [t 
Zc (ey ak)” 
(b) fz(z) does not tend to a Gaussian density. The central limit theorem does 
not apply, because V[X;] = oo V k (and, actually, E[X;,] does not exist). 

a) Y/2; (b) 1/2. 
40. (a) 1/8; (b) 7/144; (c) ~ 0.9667. 


for zER 


fa(z) = 


364 Appendix B: Answers to Even-Numbered Exercises 


(b) e1; (c) 1/¥%; (d) 
4. (a) 
Oify <a 
Fymlo ln) yee 
lify>1 
(b) (1 + 29); (c) 1/45; (d) 1/30. 


+42 
46. (a) 3/4; (b) 13/24; (c) 0; (4) XP + 3.X1 + 5. 
48.1—e7} (0.6321). 


1 (Ina — 15)? 
FX1X90-Xg0() & aes exp {OEE for 7 >0 


54. (a) X) + Xq + 4; (b) 4(X1 + X2 + Xs); (c) 1/18. 


Chapter 2 
2. (a) p(1 — p); (b) 0 ifn Amand 1 ifn= 
4. t/(2x7) 
6. The increments are not independent, but they are stationary. 
8. (a) 


b) E[X(t)] = 1/(1 +t) and Rx (ti, te) = E[X(ti + t)]. 

10. No, since B[X(t)] (= (1 — e~*)/t) depends on t. 

12. No, because E[Y(t)] (= t) depends on t. 

14. B[X(t)X(t+s)] = E[Y%t(t+s)] =t(t+s)/7 £ Rx(s). Consequently, the 
process is not WSS and therefore not SSS. 

16. Sy (w) = 2(1 — cosw) Sx (w). 

18. h(0)q(t). 

20. Cx (ki, ke) = Cx (ke — ki) = 0 if kg —ky 4 0 and Cx (ko — ki) = p(1 —p) if 
ko — ky = 0. Since mx (k) = p, the process is WSS. We have that C’x (0) < co 
and limyx| 00 Cx (k) = 0. It follows that the process is mean ergodic. 

22. Yes, because C'x (0) = Rx (0) =1 < ce and limjs) 00 Cx(s) = 0. 

24. ¢/2T. 


26. (a) e7# *. (b) (Y,Z) ~ N(py = Oz = sop = 2107 = 2) p= 0). 
28. (a) and (b) No, because BIY (t)| (= e~*) is not a constant. 

30. m(yo; to) = yo/2 and v(yo; to) = yé- 

32. (a) N(O,2); (b) 2e~88; (c) 4(1 + e~8). 

34. (a) N(0, 1); (b) U(1, 2). 
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Chapter 3 
2. (a) 1/4; (b) (i) 0; (ii) no, because E[Y(t)} (= t/2) is not a constant; (iii) 1/2. 
4. 11/12 
6. 2/3. 
8. 0.1465 
10. 
1/41/21/4 
1/21/2 0 
1/4 1/2 1/4 
12. 2/3 


14. (a) Aperiodic, because it is irreducible and po,9 > 0; (b) 
i 
p et | 
m=(=) (| —) fori =0,1,2,3,4 
: (2) (ee 


22. (b) g—pif p < 1/2; 0 if p = 1/2; p—q if p > 1/2; (c) 0 if p < 1/2; 
(q/p)*- [1 — (q/p)], for k = 1,2,..., if p > 1/2. 
24. (a) A single class, which is recurrent and aperiodic; (b) 7; = 1/5, for all 4; 


(c) no. 
28. (b) 
1 1 
(k +1)! (4) 
4a 2 1 


= Aen > ats! fo Ae 3 


32. (a) 1 (because the chain is irreducible and po,9 > 0); (b) mo = 3/7,m71 = 
2/7, = m3 = 1/7; (c) (i) 20.1716; (ii) 1. 


34. (a) P™ = P@), where 
1—pOp 
P®=/ 0 10 


1-—pOp 


30. (a) O0< a <1; (b) 


To 


if n is even, and P™ = P if n is odd; (b) d = 2; (c) (i) mo = (1 —p)/2; (ii) no, 
because this limit does not exist (since d = 2). 
36. (a) m = 2/3 and 7; = (1/4), for 7 =1,2,...; (b) 1. 
38. (a) TM = 73 = 1/6, 7) = 72 = 1/3; (b) d = 2; (c) 2/3. 
40. (a) (i) 
1 0 0 0 O 
1/41/21/4 0 0 
0 1/41/21/4 0 
0 0 1/41/21/4 
0 0 0 0 1 
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(ii) yes; ™ = 7g = 1/2 and 7¥ = 0, for j = 1, 2,3; (b) 1/4. 

42. (a) 0<a<1,0 <6 <1; (b) m = m2 = 2/5 and m = 1/5; (c) (i) 2/3; 
(ii) 1/4 + a/2. 

44. (a) a #1, B £0; (b) m) = 1/9, 7, = 5/9, and m2 = 1/3; (c) a = 1/2 and 


B =1; (d) 3. 
46. 
eee) amen fe 
i ee eye arte 
and 
2/3 if j =0 
lim pf") = lim pf) =< 0 ifj=1 
00 % n—0o 3 “fo 
1/3 if j = 2 


48. (a) 1/2; (b) we have 
2(7$)* + (15)? — 4p +1 = 0 


We can check that the value (1/2)? = 1/4 is not a solution of the equation, 
so that 74 4 72. 


50. 
: (ny) _ flifj=1 
pain Pig = ‘0 otherwise 
and 

2/5 if j = 0 
: (n) _ }. (n)_ }. (n) _ 0 ifgj=1 
gH Pog a Pag = eg bah yo 
2/5 if j = 3 


52. (a) -1 + V2 (~0.4142); (b) for example, po = po = p3 = 1/3 and p; = 0. 
54. (a) The state space is {0,1,2}, and the matrix P is given by 


(1 — pi1)(1 — pe) pi + p2 — 2pip2 pipe 
1l-—p p 0 
(1 —p1)(1 — pe) pi + po — 2pip2 pip2 


where p; := e~‘, for i = 1,2, and 


— A2 At 
eas 5 OOO WE c5 ere e 


(b) Yes, by the memoryless property of the exponential distribution. We cal- 
culate 
(1 — p1)(1 — pe) pi(1 — p2) (1 — pi)p2 pipe 
p= 1— P1 P1 0 0 
1—p2 0 2 0 
(1 — p1)(1 — pa) pi(1 — pe) (1 — pi)p2 pipe 


re 37 = 9 possible states: 0 = (0,0), 1 = (0,1), 2 = (0,2), 
1,1), 5 = (1,2), 6 = (2,0), 7 = (2,1), 8 = (2,2). We find that 


Ww 
II 
— 
— 
fw) 
— 
xe © 
ll 
as & 
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2pp 0 p00000 
pppOopdo00dvdd 
0 p2p00p 00 0 
p00 pp0 p00 
P=|0p0p0p0p0 

O00 pO0ppO0O0p 
000 pO002pp 0 
O000p0 ppp 
000 00p 0 p2p 

where p = 1/4; (b) 7; = 1/9, for all 4. 

58. n. 

60. (a) PLN = 1] = (1/2)""! and P[N = i] = (1/2)"~**!, for i = 2,3,... ,n; 

(b) (n= 1)/(2n); (c) Gla =n —1,4= 2). 

62. (a) 


1/N1/N1/N...1/N  1/N  1/N 
0 2/N1/N...1/N  1/N  1/N 
PS hon wha ate hee aks en odie 
0 0 0... 0 (N—1)/N1/N 
O> 70). 20 aaa “0 0 1 


(b) (i) We have 
a 1 ; 
m= lt mit 7 (mist +... + my) fori=1,...,N—1 


We find, with my = 0, that m; = N; (ii) m; = E[Geom(p := 1/N)] = N, for 
i=1,...,N—1, 

64. (a) ~0.00002656; (b) ~0.8794; (c) ~—$20. 

66. (a) 


-4| 1/20 0 01/2 


23 1/2°.0- 0.4/2 
P=-1 1/2 0 1/2 
0 1/2022 


1 178 9-192 


(b) m4 = 7_2 = 1/16, t_, = 1/8, and mp = ™ = 72 = 1/4; (c) 3/8. 


68. mj =p; V7. 

70. (a) 9/4. 

(2.(c)j2"— 

76. (a) 30/37; (b) 16/37; (c) 4/19. 

78. (a) ~0.1170; (b) 20.0756. 

82. (a) 1 +At; (b) mo = 1 and mp = 0, for k = 1,2,...; (c) (At)?/(1 + At)?. 
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84. (b) e~44(e?4 — 1); (c) the 7;’s do not exist, because {X(u),u > 0} is a 
pure birth process. 

86. (a) a = 0; (b) mo = 10/27, 7 = 8/27, and m2 = 9/27. 

88. (a) No, because the time 7 that the process spends in eather state does not 
have an exponential distribution, since P[r > t] = 5e~* +2 e~2t; (b) +0.2677. 
90. 12(1 — e~**) /(3 — 2e7?). 

92. (a) Ao = 2X and An = Mn = A, for n = 1,2,...; (b) the 2;’s do not exist. 

94. (a) Xo = 2A, Ay = A/2, wr = 3A/2, and pg = 2A; (b) 1/2; (c) mo = 3/8, 
my = 1/2, and m2 = 1/8. 
96. 

c—k 


k c—k 
Kear ( Nem HAE 4 1) (1 a ewer) 


98. (a) We define the states 


0: the machine is functioning 

1: failure of type 1 

2: failure of type 2; first repairing operation 

3: failure of type 2; second repairing operation 


(b) u/[u + (2—a)A). 
Chapter 4 


2. (a) 0; (b) t+ min{t?, t+7}+min{t, (t+7)?}+8°; (c) (i) yes; (ii) no, because 
Cov{X (t), X(t + 7)] depends on ¢; (iii) no, since Cov[X(t), X(t + 7)] 4 o7t; 
(d) 

t/2 if0<t<1 
PB(t),B(t?) = t71/2 ift>1 


(a) No, because X(t) > 0; (b) no, since E[X?(t)] (= t) is not a constant. 


4, 
Brownian motion with o? = 2. 

(a) No, because Y(t) > 0; (b) 1; (c) 2s/t; (d) no, because Cov[Y(s), Y(t)] 
is not a function of | t — s |. 
12. (a) &(j + 1 — i) — (j — i), where @ is the distribution function of the 
N(0,1) distribution; (b) ~0.4659. 
14. (a) 


4. 
6. 
8. 
10. 


1 
a (t+e~min{t + ¢,t + s}) 


(b) (i) yes; (ii) yes; (iii) no, because Cy (t,t + s) # o7#; (iv) yes. 
16. (a) N(0,2(1 + e~)); (b) (i) E[U(t)] = 0 and V[U(E)] = 2(¢+ e7* — 1); 
(ii) yes. 
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18. (b) No, because we have nas EMG )] = exp{(u + $07)t}, which depends 
on ¢ (if w # —$0°); if w = —40?, then E[Y(t)] = 1, but E[Y(t + s)Y(t)] 
(= e?¢) is not a function of s. 

20. (a) 0; (b) yes, because X and Y are Gaussian random variables and their 


covariance is equal to zero. 
22. E/Y (t)| = 0 and 


o* e(s+t) clt—s| 
Cy(s,t) == (e —e ) for s,t > 0 


24. (a) N(0, 11/3); (b) 


3V3 5/2 3 
sous atl > 0 
fr, (t) Tae exP | — 5 for t > 


3 


26. (a) E[Y(t)| = 0 and 


3 2 = 2 
Cov[Y (t), Y(t + s)] = * +e (s- as ) ) 


(b) (i) er (ii) no, because Cov{Y (t), Y(t + s)] depends on ¢; (iii) no, since 
Cov[Y (t), Y(t + s)] A t[1 — (+ 8)]; (c) & 2P[N(0,1) > V12d]. 
28. We Bnd that 


2d Pet \d a ae, 
= ————— > 
Fro) = ~ Fae exp = =} dt (===) ee 
30. (a) 0; (b) 1+ 2t; (c) no, because Cov{Z(t), Z(t + s)] depends on ¢; (d) 


ae = 2 
fryz)(t) = . eee P{-jaaas for t >0 


Chapter 5 
2. (a) e~?-® (0.0821); (b) ~0.0028; (c) 1/3; (d) 
10 
Cx (tr, ta) = a min{t,, to} 
12 


4. (a) (i) e~? a 1353); (ii) 20.1839; (iii) ~0.3233; (b) no, because E[Z(t)] = 


exp{t(e~! — 1)}, which depends on t. 
6.e~° 

8. As. 

10. (a) (1 — e7!)?; (b) 1.64. 

12. (a) <0 3535: (b) 0.0119. 
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14. (a) 3A/64; (b) ¢? 

16. (a) (i) Mean = 15, variance = 240; (ii) mean = 10, variance = 520/12; (b) 
~14.785. 

18. 1/(2V). 

20; PIN* (0) = 1} =‘o(8); 

22. (a) Let y := A(e’ — 1). We find that 


2y 1) 
nl) ta x0 
Mnu+s)(s) = is Dye * 
1 ifs =0 


b) E[N(t + S)] = A(t +1) and V[N(t + S)] = A(é +1) + §A?. 

a) e~*(b—a)/15; (b) 1/16; (c) 1/16. 

. (a) 0.6723; (b) 1/2; (c) 1/3. 

a) U(0,s); (b) 5e~* (~0.0916); (c) 10/3. 

a) (i) V/(1 —t), for 0 <t <1; (ii) 1/2; (b) (ii) P[N(0,1) < 2] (0.9772). 
a) 0.5590; (b) 4.75; (c) 0.0409; (d) ~0.0328. 

i. 0.9817. 

36. 1—e7? (14+ 2A 4 3A?). 

38. ~0.217. 

40. (a) n!/n”; (b) 2(1—t) if0 <t <4, and 2(t-$) if} <t<1. 
42. {X(t),t > 0} is a Poisson process with rate A. 

44. Yes; we have that p,; = 0 if 7 < i and 


ne 
5 ( 
0. ( 
2. ( 


\vi-vi 
(Vi— vil 


-X 
Pig =e 


if 7 >i, where i,j € {0,1,2, 4,9,...}. 


46. (a) 1/3; (b) (i) 2x e~®’ for x > 0; (ii) ~0.45; (c) 9/5. 
48. (a) ~0.2565; (b) 1/5. 
50. (a) 0.56; (b) 3/16. 
52. (a) 1- 2e-1 (0.2642); (b) 0.75 t + 0.5 ¢?. 
54. (a) 1/3; (b) 588; (c) 232. 
56. (a) ~0.4562; (b) 2 if ) 2; (d) 1/2, if s € (0, 1] or (2, 3]. 
58. (a) In(s? + D/ in| P) ~0.6068. 
60. (a) 12¢/(1 — e~*); (b) P[N(0,1) < 0.69] (0.7549). 
62. (a) Mean = 8 ot ee = rh (b) 401; (c) 
0 ifs <6 
eS: 6p e4 


P[S <s|N(8)-N(6)=1]=4 4, °9 
S-Ditt<s<s 


1 ifs>8 


64. (a) (i) Mean = variance = 21; (ii) mean = variance = 24; (b) ~0.1562. 
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66. (a) A1/(2A1 + Az); (b) Ait; (c) 1 — Ze7? (0.9323). 
68. (a) G(a@ = 2, A); that is, 


fr(t) =d?re~*”_— for r > O andi =1,2,... 


(b) we find that 


7 (A? (At)2ntt 
P{N(t) =n] =e"** ‘Qayt * (nt Dy! for n =0,1,... 
(c) 3/(2A). 
e (a) w/[m + (2 — a)Al; (b) (8/u)(2 — a)/(3 — a). 
2. (a) G(a@ = 2, A); (b) 0.81; (c) 21.17. 
v4 (b) ~1/2; (c) 1/2. 
6. (a) We find that 
my(t) = exp {a} (: pelts 5) -1 for1/2<t<1 
(b) P[N(O, 1) < 0.48] (~0.68); (c) 5/8. 
78. (a) We calculate 


1 t 
mutt) = 2 (ex {£4} —1) forO<t<1 


(b) e — e/? (~1.07). 

80. No; since the number of events per time unit increases linearly, the time 
between two consecutive events is (on the average) shorter and shorter, so 
that the random variables 7, are not identically distributed (they are not 
independent either). 

82. i> 0. 

84. (a) 1/(2a? + 1); (b) 0. 


Chapter 6 


2. (a) N = 11/15 and Ng = 4/15; N #4 Ng +1 because 7m 4 0; (b) 1/15. 
4. (a) N = A/p and Ng =e~/# + A—1; (b) 1+ Ke =e 1), 
6. (a) We have 


state 7 departure rate from j = arrival rate to j 


0 2A 7 = ETM 
1 (2A + wt) wm = m2 + 2A 70 
2 (A + fe) mo = fh 73 + 20 7% 
k (> 3) (A+) Ty = Tea + A Wee 
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(b) 1/2; (c) 3/8. 
8. (a) We have 


state 7 departure rate from 7 = arrival rate to j 


0’ ° 10 7 = 5 70 
0 (10 +5) m = 10 7645 (m1 + 72 +73) 
1 (10+5) 7m =10 7% 
2 (10+ 5) m2 = 10 m 
3 5 73 = 10 m7 


(b) my =1/7, m9 = 2/7, 1, = 4/21, me = 8/63, and 73 = 16/63; (c) 5/21. 
10. (a) m = (n +1)(1/2)"*?, for n =0,1,...; (b) 10/11; (c) 0.4180. 
12. (a) tm = (x45) (xta)”, for n = 0,1,...; (b) A (A+ pH): 

14. 4/(u +4). 

16. (a) We have 


state 7 departure rate from 7 = arrival rate to 7 


0 A To = HA ™, + He TM, 
1h (A+ p1) T1, = pid To + He M2 
i) (A + pa) M1, = (1 — pi)A mo + Hi Te 
2 (A + fa + pa) M2 = A (71, + 712) + (i + M2) 73 
sone l2 (A+ pf + M2) TM = A Mp1 + (Hi + H2) Trt 
13 (1 + M2) 713 = A TMH2 
(b) the probability requested is given by 
; P 12 
1 
———— |p) 70 + TM, + T 
(1 — 7713) |» eee Gan PB : 


18. (a) We find that 


aN 
and In = (2) gn—1 0 for n = Vg2y ses. 


(b) (A+) 71, = pw 7; (c) the fraction of time that the second counter is 
open is given by 
d?(2u — 2) re d? 
2A + w)(2Qu +A) — w(2e+ A) 
20. (a) We have 


state 7 departure rate from j = arrival rate to j 


AT = M1 
(A+ 2) m1 =A 70+ 2p 72 
(A + 2p) hq =A M+ 2p 713 
2p 73 = AT 


WNrF © 
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We find that if \ = 2, then mp = 1/7 and 7, = 72 = 73 = 2/7; (b) 1.2; (c) 2. 
22. (a) We find that 


n m 
Tinym) = a as ma is for alln,m > 0 
y 36 36 48 48 


(b) ~5.41; (c) 1/5. 
24. (a) We define the states 


0: the system is empty 

n: there are n > 1 ordinary customers in the system 

ns: there are n > 1 customers in the system and the 
special customer is being served 


The balance equations of the system are the following: 


state 7 departure rate from 7 = arrival rate to j 


0 (A+As) To = m1 + Us ™1, 
1 (A+As +) m1 =A To + 2 Te + ps Te, 
1, (A+ fs) 1, =As To + bt M2, 
n>2 (A +As + 24) tr =A Tr-1 + 2H Tn41 + Ms TM(n41)s 


1s (n 2 2) (A+ ps +H) tm, = As Ta-1 $A Mn-1), + HT (n+1)s 


(b) we define the states (m,n) and (m;,n), where m € {0,1, 2} (respectively, 
n € {0,1}) is the number of customers in front of server no. 1 (resp., no. 2) 
and s designates the special customer; there are therefore eight possible states: 
(0,0), (1,0), (0,1), (15,0), (1,1), (15,1), (25,0), and (2,. 1); (c) we can use the 
state space of part (b); we find that the probability requested is given by 


( As ) 7(1,0) +( As ) ( vi ) (1,1) 
L+As/ Tao) + 71,1) H+As) \H+As/ M10) + 70,1) 


nm 
26. (a) m = 1- om and 7, = (4 wo, for n = 1,2,... (this model is 
equivalent to the M/M/1 model with service rate equal to 2 rather than j.); 
(b) 1/3. 


28. 7 = 1/4 and m = m2 = 3/8. 

30. (a) We define the states (i, 7), for 7,7 = 0,1, where i (respectively, j) is 
equal to 1 if a customer is being served by the hairdresser (resp., the assistant), 
and to 0 otherwise; (b) the balance equations of the system are the following: 


state j departure rate from 7 = arrival rate to j 


) (Ai + Az) 70,0) = #1 71,0) + #2 70,1) 
) (Ar +A2 + wi) 71,0) = At 70,0) + #2 7,1) 
) (Ai + #2) 70,1) = A2 70,0) + Ha 71,1) 
) (M1 + Me) M11) = Ar Toy + Ar +A2) 7,0) 
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(c) (i) N = 70,1) + (1,0) +2 7a,1)3 (ii) T = N/d_, where 
Ne = At (1 — (4,1) + A2 (77(0,0) + 74,0) 

32. (a) We define the states 


0: the system is empty 
1,: there is one customer in the system, being served by server no. 1 
lg: there is one customer in the system, being served by server no. 2 
2: there are two customers in the system, one per server 
29: there are two customers in the system, being served by server no. 2 
3: there are three customers in the system, two of them being served 
by server no. 2 


(b) the balance equations of the system are given by 


state j departure rate from j = arrival rate to j 


0 A To = fi 1, + He (71, + 725) 
14 (A+ p11) ™1, =A T+ pe (m2 + 73) 

lo (A + He) 71, = M1 7 

2 (A + pi + fe) 72 = A (71, + 7™M1,) 

22 (A+ 12) Te, = fi 73 

3 (1 + M2) 73 = A (m2 + 722) 


(c) the fraction of time requested is (72, + 73)/(1 — 7 — ™71,). 
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